Open Devices Corner

Flashing AOSP on Xperia XZ

Fri, 19 Oct 2018 22:26:47 +0200

The builds for the Xperia XZ will no longer be updated. This guide is severely out of date (as of 2022). The builds should still work, but proceed with caution.

Follow these steps closely. Do not leave out a step and do them in the correct order.

A big thanks to StaticGTF for writing large parts of this guide!

TL;DR

For people who know what they are doing:

Flash latest stock image, then unlock bootloader
Download v9 binaries from developer.sony.com, Flash binaries image via fastboot to oem partition
Flash this tweaked TWRP recovery, credits to Artem Labazov(@ab123321)
Flash latest build from sx.ix5.org from inside TWRP

That’s it.

Detailed Steps

1. Computer Setup

In order to install this ROM, you will need a working computer environment to install the latest FTFs, recoveries and the OEM binaries.

There are numerous other guides out there on how to set up fastboot, but this should be enough for most.

On Windows: Download and install the following components to your computer:

Minimal ADB and Fastboot from xda: Minimal ADB and Fastboot
Androxyde’s FlashTool from: xda: FlashTool

You will also need to install these drivers for your phone:

Regular Xperia XZ driver
Flashmode Driver
Fastboot (S1) Driver

The drivers are located in the FlashTool installation folder. You will probably need to reboot your computer with disabled driver signature verification in order to install these drivers.

On Linux: Download Androxyde’s FlashTool from: xda: FlashTool.

Then, install the fastboot and adb tools via your distro’s package manager.

On Ubuntu: apt install android-tools-adb android-tools-fastboot
On Arch Linux: pacman -Syu android-tools.

2. Download necessary files

Make sure you have all the files needed before starting this process. Best proceedings practice should mean that you’ve got all images located in a folder where you can easily flash them to the phone with least amount of typing (same folder as fastboot.exe).

Download the .FTF file for the latest stock firmware. It should should be something ending in .184 or .192 for your region. See this list of latest stock firmware files. You can also use the tool XperiFirm for downloading, which is included with FlashTool.
Download this tweaked TWRP recovery, credits to Artem Labazov(@ab123321)
Download Sony OEM binaries for tone on Android Pie, Kernel 4.9 from the Sony Software Binaries page.
The filename will be SW_binaries_for_Xperia_Android_9.0_2.3.2_v$VERSION_tone.zip, where $VERSION is currently v9.
Download the latest AOSP 10 build from sx.ix5.org
Optional: Download the latest stable Magisk from xda: Magisk
Optional: Download OpenGApps from opengapps.org. You need to chose Android 10.0, ARM64. You can choose any size variant, I would recommend the pico one because it has the least bloat.

3. Unlock Bootloader

You need an unlocked bootloader. Read carefully what this means.

If your bootloader is already unlocked, you can skip ahead to the next step.

Warning!

Back up your Trim Area("TA") partition before unlocking the booatloader. You cannot undo an unlock and your DRM keys, possibly along with your warranty, will be lost forever!

If you are on a locked bootloader you will need to unlock your bootloader using this page: Sony Developer - Unlock bootloader

4. Flash latest stock firmware

The following steps are done from your computer with your device attached via USB-cable.

If you have previously upgraded to the latest stock firmware(.192) and only tested out e.g. OmniROM, there is no need to re-flash the stock firmware and you can skip this step.

You can access S1 flash mode by powering down the device completely, then holding VOLUME-DOWN while connecting the USB cable. The notification LED should turn green and you can stop holding the volume button.

Flash the latest available Oreo .FTF stock firmware using Androxyde’s FlashTool.
(Guide for flashing FTF with FlashTool: xda: How to use Flashtool)

5. Flash TWRP Recovery

You can access fastboot mode by powering down the device completely, then holding VOLUME-UP while connecting the USB cable. The notification LED should turn blue and you can stop holding the volume button.

Flash the TWRP recovery from the command line:

fastboot flash recovery twrp-3.2.1-0-kagura-ab-x.img

(You need to use this tweaked recovery because the official TWRP builds can not mount /data or handle encryption).

6. Flash OEM binaries

Extract the Sony OEM Binaries zip-archive that contains the image-file needed for the next step. Flash Sony’s OEM binaries with this CLI command: (Note that the destination differs between the recovery and oem!)

fastboot flash oem SW_binaries_for_Xperia_Android_9.0_2.3.2_v3_tone.img

7. TWRP Recovery operations

Remove the USB cable from phone and check that it is completely powered down. Then boot up your device to recovery by pressing and holding VOLUME-DOWN and the POWER button at the same time. When the white Sony logo appears, release all buttons.

7.1 Wipe data (optional, but highly recommended)

Wipe /data, /cache and Dalvik cache in TWRP:

When booted into TWRP, swipe from left to right to enable system modifications. Do not proceed with system partition marked as “Read only”!
Click “Wipe” and choose to perform a factory reset. Swipe to proceed. (This will keep the files on your internal storage but delete all your app data).

If you are coming from stock firmware, you absolutely need to do this. You might be able to skip this step if you are upgrading from a previous Pie or Q build, but if you run into problems you need to at least wipe the caches, if not `/data` as well.

7.2 Flash the AOSP ROM

When booted into TWRP, swipe from left to right to enable system modifications. Do not proceed with system partition marked as “Read only”!
Connect your phone to your computer via USB. The Phone should appear in your devices-list to the left in Windows Explorer.
Transfer AOSP ROM zip-file to the phone’s internal storage.
Click “Install”-tile and select the AOSP ROM file
Swipe from left to right to start the flashing procedure.

7.3 Fix /dsp file labels

This only for Android Pie builds. Android 10 does not need this fix any more.

If you ever installed an Android Oreo-based custom ROM - e.g. OmniROM 8.1 - your SELinux file labels for the dsp partition will be wrong. For a more detailed explanation and Download, see File relabeling for SODP.

Simply flash the DSP label fixer zip file in TWRP.

7.4 Flash dualsim patcher (optional)

If you have a dualsim device, download the DualSim patcher.

Transfer the DualSim patcher zip file to the phone’s internal storage.
Click “Install”-tile and select the DualSim patcher
Swipe from left to right to start the flashing procedure.

7.5 Flash OpenGapps (optional)

If you have problems afer flashing GApps, wipe all data and flash the ROM again, without GApps.

Transfer the OpenGApps zip file to the phone’s internal storage.
Click “Install”-tile and select the OpenGApps zip file
Swipe from left to right to start the flashing procedure.

7.6 Flash Magisk (optional)

Flash Magisk v18.1 or later. Then install the Magisk Manager app. But read carefully which modules are compatible!

Transfer the Magisk zip file to the phone’s internal storage.
Click “Install”-tile and select the Magisk zip file
Swipe from left to right to start the flashing procedure.

If you are using Magisk, your phone will restart once on first boot. This is fully normal and is to be expected.

8. Reboot

Go back to the TWRP home screen and select “Reboot”. If TWRP prompts you to install its own app, decline and choose to restart normally.

Your phone will take a good while to finish booting up (~5 minutes or more). Leave it alone until done.

Attention!

Please also read the full bug list!

Optional Goodies

You can try OpenCamera, Snap Camera(from LineageOS), or one of the numerous Google Camera ports.

Troubleshooting

If your phone can’t boot past the white Sony logo or freezes on Android boot-animation, try to simultaneously press POWER + VOLUME-UP until the first vibration. This will reboot your phone.

If you need to re-enter TWRP, your phone must first be powered off. You can always force this no matter where and when by long-pressing POWER + VOL-UP until you feel three vibrations. Then release buttons and boot to TWRP with POWER + VOLUME-DOWN.

Magisk

If you cannot boot and have Magisk modules installed, it might be that they are incompatible with this ROM. Before reporting a bug, please remove all Magisk modules(by flashing the Magisk uninstall zip) and try booting again.

Encryption does not work

See “Formatting /data” below.

Formatting /data

If you still cannot boot the AOSP builds, try formatting your /data partition.

If you ever had encryption turned on while running the stock firmware, you also might have to format your internal storage. This is because the stock firmware seems to be using a different encryption version or algorithm than the AOSP builds. The stock firmware runs encryption by default since Android Nougat.

Back up all your photos, apps etc. to your computer as this will wipe your internal storage completely!

For 32Gb (single-SIM) devices, use this command:

fastboot format:ext4:0x58dffc000 userdata

For 64Gb (dual-SIM) devices, use:

fastboot format:ext4:0xcd5bfc000 userdata

You need to use the fastboot method because wiping in TWRP does not leave enough space for the crypto footer (needs to be 32Mb), which you need to run Full Disk Encryption (FDE).

Method for calculating needed hex values for fastboot format, including space for the crypto footer:

# Single-SIM:
# Reserve space for data encryption (23857201152-16384)
BOARD_USERDATAIMAGE_PARTITION_SIZE := 23857184768
# Dual-SIM:
# Reserve space for data encryption (55125737472-16384)
BOARD_USERDATAIMAGE_PARTITION_SIZE := 55125721088

Then use python to calculate the hex values:

$ python
hex(23857184768)
 '0x58dffc000'
hex(55125721088)
 '0xcd5bfc000'

A big thanks to StaticGTF for writing large parts of this guide!

Building Android

Fri, 19 Oct 2018 22:14:54 +0200

The builds for the Xperia XZ will no longer be updated. This guide is severely out of date (as of 2022) and will most likely no longer work. Proceed with caution.

This guide will asume you follow the official Sony build guide, but add some extra tips.

Initialize your build directory

Create a ~/android/build-env folder, cd into it.

Using the repo tool

Download and install the repo tool, either via your package manager or directly from Google. Instead of copying the file, it’s better to symlink:

mkdir -p ~/.local/bin
# Add ~/.local/bin to PATH
echo "export PATH=$HOME/.local/bin:$PATH" >> ~/.bashrc
# Symlink
ln -sf ~/android/.repo/repo/repo ~/.local/bin/repo
# Check:
readlink $(which repo)
# -> /home/builder/android/.repo/repo/repo

Then, run the repo init command:

repo init -u https://android.googlesource.com/platform/manifest \
  -b android-10.0.0_r39 -g default,-x86,-mips,-darwin,-notdefault

Use the appropriate branch tag instead of _r39. For an overview of currently published tags, see Codenames, Tags and Numbers. Most often, you’ll want to pick the latest Pixel tag, e.g. the Pixel 4.

Ubuntu chroot

Since Google use Ubuntu as a reference build environment, it is advisable you use it as well to avoid incompatibility problems. If you're on Ubuntu already, you can skip this step.

If you’re on another system like Fedora or Arch Linux, you can set up a Ubuntu chroot build environment instead.

Another alternative to nspawn is to use docker, see Sony AOSP on docker.

A build chroot takes about 1-1.5GB of space and will save you from headaches, e.g. when Android modules aren’t compiled against your host glibc.

Install debootstrap, then run this (bionic is the ubuntu version codename for 18.04 LTS):

debootstrap --variant=buildd --arch=amd64 bionic ~/android/ubuntu-android

chroot into the newly created builder system, either via chroot or just using systemd-nspawn (Assuming you created your android build environment via repo init in ~/android/build-env):

sudo systemd-nspawn \
  -D /home/<your-username>/android/ubuntu-android
useradd -m -s /bin/bash builder
# -> No need to add a password, nspawn will log you in automatically

For Android Q, the way .img files are mounted during build has changed, e.g. resize2fs fails with a standard nspawn container.

ln -s /proc/self/mounts /etc/mtab

And add this to the options of systemd-nspawn:

    --bind=/dev/loop0:/dev/loop0 \
    --bind=/dev/loop1:/dev/loop1 \
    --bind=/dev/loop2:/dev/loop2 \
    --bind=/dev/loop3:/dev/loop3 \

(So that enough loop devices are available)

Install required packages

apt install software-properties-common python
add-apt-repository universe
apt update
apt install bison g++-multilib git gperf libxml2-utils make \
  zlib1g-dev zip liblz4-tool libncurses5
apt install openjdk-8-jdk
# Might be needed on Android 10:
apt install flex

# The following dependencies are all optional:
##############################################
# Faster recompilation
apt install ccache
apt install rsync libssl-dev aapt
# Kernel recompilation:
apt install bc autoconf automake autopoint autotools-dev bsdmainutils gawk \
  groff-base libarchive-zip-perl libpipeline1 libtimedate-perl libtool kmod
# For selinux:
apt install setools python-networkx policycoreutils

Cross-reference the needed packages with the official Sony build guide should this document be out of date.

If you’re building in a chroot: Exit out of the build system by typing exit (and pressing Ctrl-] three times for nspawn), then re-launch your build system and log in as your builder user. If you want to share your own ccache etc. with your build environment, bind the appropriate directories.

sudo systemd-nspawn \
--bind=/home/<your-username>/android/build-env/:/home/builder/build-env/ \
  --bind=/home/<your-username>/.ccache/:/home/builder/.ccache/ \
  -D /home/<your-username>/android/ubuntu-android \
  --user=builder

Local manifests

Clone the Sony local_manifests git repo as instructed. For legacy devices, meaning loire and tone, you need to checkout the android-10_legacy branch of local_manifests. For recent devices have to use the master branch.

git clone https://github.com/sonyxperiadev/local_manifests .repo/local_manifests
cd .repo/local_manifests
# For recent devices:
git checkout master
# For legacy devices
git checkout android-10_legacy

Applying needed patches

Run repo_update.sh. Your build directory will now be ready for a first build.

Felix: If you want to recreate my own builds, use


git clone https://git.ix5.org/felix/local-manifests-ix5 local_manifests -b 'ix5-customizations-10'

instead. After you run repo_update.sh, also run q_repo_update.sh. This will fetch my newest changes as well.

Prepare build environment

Run source build/envsetup.sh inside your build environment and then lunch aosp_<product>-userdebug.

To find out the product name of your device for aosp_<product>-userdebug, see the Sony Devices overview. E.g. for a single-SIM (“SS”) Xperia XZ, choose aosp_f8331, for a dual-SIM XZ, choose aosp_f8332.

You can also run lunch without any arguments to get a list of all available build targets. They will look like aosp_f83xx-... where f83xx is the model number of your device. You can choose between eng and userdebug builds, see Choose a target.
For your own usage, userdebug is most likely what you want.

The user target is meant for public releases and doesn’t include root. You’ll need to look into vendorsetup.sh (AndroidProducts.mk for Android Q) - and perhaps add it for your device - if you want to use it.

Build kernels

On Android Q, you need to use a script to build the kernel images. Navigate into kernel/sony/msm-4.9/common-kernel and run build-kernels-gcc.sh.
If building for a Kernel 4.14 device, go into kernel/sony/msm-4.14/common-kernel and use build-kernels-clang.sh instead.

Edit build-kernels-{gcc,clang}.sh and change the list of platforms in the script so that it looks like this:

# Override:
TONE="kagura"
PLATFORMS="tone"

This will save you from building the kernel for all Sony devices and only build for the Xperia XZ (“kagura”).

Build & Flash

For legacy non-A/B devices like the Xperia XZ, run this command:

make bootimage systemimage

Newer devices require building for more partitions, like vendor, dtbo, product, vbmeta and others.

In case your build stops at about 90% with this error, most of the time you should be fine by just restarting the build with -j 1¹.

[ 93% 12745/13686] //frameworks/base:hiddenapi-lists-docs Metalava [common]
FAILED: out/soong/[...]hiddenapi-lists-docs-stubs.srcjar

However, if this still fails, try cherry-picking this patch from LineageOS. See also this reddit post

Then, use fastboot to flash the generated images

cd out/target/product/kagura
fastboot flash boot boot.img
fastboot flash system system.img

For 4.14 devices, also flash the other partitions like vendor, dtbo etc.

Do not forget to flash the appropriate Software Binaries to the oem partition as well!

Optimize the build

A full build will take about two to three hours² on a beefed-out recent system (Core i7 8th gen 4 cores, 16GB RAM, good SSD). That time can however be cut down for successive builds to around an hour by utilizing ccache and parallelizing the build.

It is recommendable to use a ccache size of around 50-100GB, depending on whether you plan to build different ROMS or for multiple devices.

# On Android Pie and before:
_CCACHE_EXEC=prebuilts/misc/linux-x86/ccache/ccache
# On Android Q:
_CCACHE_EXEC=/usr/bin/ccache
# Set ccache size:
$_CCACHE_EXEC -M 50G

Then set the environment variable with export USE_CCACHE=1. This variable needs to be set anew every time you open a new terminal, so it is advisable to put export USE_CCACHE=1 at the end of your .bashrc in your home directory (or /home/builder/.bashrc if you’re in a chroot).

ccache compression might also save a lot of disk space for you. Set export CCACHE_COMPRESS=1 in your .bashrc. This cuts ccache disk usage down to about 5GB per device, but may incur some performance penalty.

On Android Q, you need to install the ccache package and set CCACHE_EXEC in your .bashrc manually to e.g. /usr/bin/ccache. The reason for this is that Google no longer ships a prebuilt ccache with Android.
For more info, see Android Q changes and build/core/ccache.mk.

To use parallel compilation, you need to find out how many threads your CPU has via nproc. Then use that number for make -j<nr-of-threads>. For most computers it will be the number of CPU cores times 2, e.g. “8” for a 4-core CPU.

You can also set export WITHOUT_CHECK_API=true in your ~/.bashrc to skip the metalava checks which take up a lot of RAM. Obviously, you should run the metalava checks when you change any Java code yourself, but you can save a bit of time if you skip them for routine builds.

If you're on AOSP Android Q, you need to manually cherry-pick Bring back env flag to skip checkapi. On LineageOS and on current (as of 2019-09-07) master, the CL has been merged already.

Distributing

If you want to share the fruits of your labor, you can create compressed flashable .zip files. Use make otapackage instead of make systemimage [...]. This will create a flashable zip file in out/target/product/<device-codename> named something like aosp_f8331-ota-eng.hostname-2018-10-19. You can flash this file via adb sideload or use a custom recovery like TWRP.

If you intend to share your builds publicly, you should generate and guard your own set of signing keys. See Signing builds for release for more information.

After you have tested your build on your own device, you can share the file freely, given that it contains free/libre software under the GPL and other free licenses. Anything GPL-licensed also demands you have to divulge your build sources. So put your patches into a git repo and share them with the world.

Speedier development

Use m module.name -j $(nproc) to only rebuild a single module. You can then adb push or even adb sync the changes directly to your device
Use make bootimage to re-generate kernel and ramdisk only (boot.img)
Use adb push or adb sync instead of flashing full images. N.b.: This is only possible with disabled vbmeta.

Going further

To show all available build targets, use make modules.

Show commands used live: make showcommands.

Good references:
eLinux: The Android Build System

Android’s metalava takes up enormous amounts of RAM. It attempts to verify the whole Android API at once and is badly designed. If it is run in parallel to other processes, it will frequently go OOM. By restarting the process, metalava will be the only one running and hopefully succeed. ↩︎
On Android Q, this time goes up to five hours for me. ↩︎

Fake Treble for Xperia XZ

Fri, 20 Dec 2019 16:55:47 +0100

This guide should help you create builds that can run Generic System Images (GSIs) for the Xperia XZ, even though it was never shipped with a /vendor partition.

We will move the “Software Binaries”, as Sony calls them, from the /odm mountpoint to /vendor/odm and move the vendor part from the system partition (/system/vendor) to the oem partition.

Pre:

Partition   Mountpoint
system      /system
             ↳/vendor
oem         /odm

Post:

Partition   Mountpoint
system      /system
oem         /vendor
             ↳/odm

Most steps in the regular Android build guide and the official Sony build guide apply here as well.

How-To

local_manifests

Clone my own local_manifests instead of Sony’s:

git clone https://git.ix5.org/felix/local-manifests-ix5 \
  local_manifests -b 'ix5-customizations-10'

repo_update

After you run repo_update.sh, also run q_repo_update.sh. Then, run treble_repo_update.sh. These scripts will fetch all the relevant patches you need to create a fake treble build.

You will also need to download the Android 9, Kernel 4.9 Software Binaries. Do not use the Kernel 4.14 binaries!

device-sony-customization

Then, create the folder device/sony/customization inside your build environment.

It needs to contain two files:

Customization.mk: (note the uppercase C!)

BUILD_KERNEL := false
# Include buildvar whether to build a fake treble build or not
include device/sony/customization/customization-noproduct.mk

customization-noproduct.mk:

# Fake treble builds for kagura
PRODUCT_FAKE_TREBLE_BUILD := true

If you want to build a vendor-only OTA package, add this to customization-noproduct.mk:

# Skip adding system to ota
PRODUCT_BUILD_SYSTEM_IMAGE := false

device-sony-odm

Clone https://git.ix5.org/felix/device-sony-odm into device/sony/odm.

Then, you need to extract the “Software Binaries” - “blobs” for short - from the official images.

When you download an image from the Software Binaries page, you will receive a .zip archive, which contains a .img file. This .img file is a sparse android image¹:

$ file SW_binaries_for_Xperia_Android_10.0.7.1_r1_v2a_tama.img
 ↳ SW_binaries_for_Xperia_Android_10.0.7.1_r1_v2a_tama.img: \
   Android sparse image, version: 1.0, [...]

Sparse Images & Loop Mount

You can un-sparse the image using the simg2img² tool. Run /path/to/simg2img SW_<name>.img swbins-unsparse.img and you will receive a regular loop image of an ext4 file system.

$ file swbins-unsparse.img
 ↳ swbins-unsparse.img: Linux rev 1.0 ext4 filesystem data, \
   [..] volume name "odm" (extents) (large files) (huge files)

You can then mount that image using loop³:

sudo mount -o loop swbins-unsparse.img /mnt/

Create device/sony/odm/blobs/ and copy the contents of the mounted image there:

cd device/sony/odm
mkdir blobs
sudo cp -r /mnt/* blobs/
# Delete ext4 metadata
sudo rm -rf blobs/lost+found/
# Don't forget the colon
sudo chown -R $(whoami): blobs/

Build it!

First, because building the kernel on Android Q from inside the Android tree is quite complicated, we use a script to build them instead.

cd kernel/sony/msm-4.9/common-kernel

Edit build-kernels-gcc.sh and uncomment/edit the Override lines so the section looks like this

# Override:
TONE="kagura"
PLATFORMS="tone"

This will save you from building the kernel for all Sony devices and only build for the Xperia XZ (“kagura”).

Then, build the kernel with patched dtb:

bash build-kernels-gcc.sh

After this has finished, you should have a file named kernel-dtb-kagura in your kernel/sony/msm-4.9/common-kernel folder.

Now, go back to your Android root dir and run the build commands:

source build/envsetup.sh && lunch aosp_f8331-userdebug
make bootimage systemimage vendorimage

In case your build stops at about 90% with this error, just re-start the build⁴.

[ 93% 12745/13686] //frameworks/base:hiddenapi-lists-docs Metalava [common]
FAILED: out/soong/[...]hiddenapi-lists-docs-stubs.srcjar

Flash the resulting files in out/target/product/kagura/:

fastboot flash boot boot.img
fastboot flash oem vendor.img

Do not flash Sony’s software binary image again now, all the needed blobs are already included in vendor.img for you!

OTA

To create a flashable zip file, use make otapackage. You can install the resulting zip file in out/target/product/kagura via TWRP or any other recovery.

adb push out/target/product/kagura/aosp_f8331-ota-eng*.zip /sdcard/

Compatible GSIs

You need to use system-as-root, arm64_ab GSIs since this is now the default for Android 10.

Use fastboot flash system <my-gsi.img>.

Since SELinux is in enforcing mode, badly-built GSIs will fail to run. You can set it to permissive by changing the commandline:

BOARD_KERNEL_CMDLINE += androidboot.selinux=permissive

or set BOARD_USE_ENFORCING_SELINUX := false in Customization.mk.

Going back to regular builds

You need to:

Unset PRODUCT_FAKE_TREBLE_BUILD in customization-noproduct.mk
Unset PRODUCT_BUILD_SYSTEM_IMAGE in customization-noproduct.mk
Revert dtsi: tone: conjure oem into /vendor in kernel/sony/msm-4.9/kernel in case you want to build non-treble tone builds again

After you've flashed your fake treble build, you need to re-flash the regular Software Binaries to the oem partition if you want to return to regular builds.

Explanation

The fake treble methodology relies on some patches on top of the SODP repos.

git scripts

Here is an example of picking git patches, see q_repo_update:


pushd $ANDROOT/device/sony/tone
LINK="https://git.ix5.org/felix/device-sony-tone"
(git remote --verbose | grep -q $LINK) || git remote add ix5 $LINK
do_if_online git fetch ix5

# git checkout 'disable-verity-no-forceencrypt'
# Change forceencrypt to encryptable for userdata
apply_commit af592265685fddf24100cbc1fdcdcb5bfd2260c1
# Disable dm-verity
apply_commit b611c8d91a374f246be393d89f20bbf3fc2ab9f7

# git checkout 'treble-odm-3'
# Use oem as /vendor
apply_commit 796ff85b93d28a301cce7bd6b3e0852a35180e04

popd

This piece of code could be simplified as:

cd device/sony/tone

git fetch https://git.ix5.org/felix/device-sony-tone \
  'disable-verity-no-forceencrypt'
# Change forceencrypt to encryptable for userdata
git cherry-pick af592265685fddf24100cbc1fdcdcb5bfd2260c1
# Disable dm-verity
git cherry-pick b611c8d91a374f246be393d89f20bbf3fc2ab9f7

git fetch https://git.ix5.org/felix/device-sony-tone \
   'treble-odm-3'\
# Use oem as /vendor
git cherry-pick 796ff85b93d28a301cce7bd6b3e0852a35180e04

So it enters the device/sony/tone repository, fetches several branches from the ix5 git repos and applies the needed commits.

For example, the treble patches for tone devices: branch - Use oem as vendor.

This patch modifies the fstab file used to mount partitions by Android, sets the size of the fake vendor partition to the size of the real oem partition, sets TARGET_COPY_OUT_VENDOR and checks whether PRODUCT_FAKE_TREBLE_BUILD is set.

Another example

Same for the needed dtsi changes: treble_repo_update - dtsi: Conjure oem into vendor.

This changes the early-mount arguments so that the kernel treats the oem partition correctly on early bootup.

This same methodology applies to the rest of the needed commits. Should they be out of date, please fetch them, resolve the git conflicts yourself, and maybe send a pull request so that others might benefit.

Not a Lawyer

You should decide for youself whether this approach falls within the purview of the blobs EULA. In the EU, you might have some more rights than in other areas so some passages of the following might be void, but I’ve highlighted the relevant parts of the EULA below:

2. END USER RIGHTS AND USE.

Sony Mobile grants You non-exclusive, non-transferable end-user copyright rights to download the Software on the local hard disk(s) or other permanent storage media of one computer and install the Software, by using the Fastboot Protocol, on a single Sony Mobile device with an unlocked boot loader at a time. Except as provided by any Open Source License Terms, You may use the Software only for the sole purpose of testing and validating Your own applications and/or content (“Purpose”). […]

3. LIMITATIONS ON END USER RIGHTS.

a) You may not use (other than as expressly permitted for the Purpose), modify, translate, reproduce, or transfer the right to use the Software or copy the Software. You may not use the Software for any reason other than the Purpose.

For more info, see source.android.com or search for “Sparse Image”. ↩︎
You should already have the tool in out/host/linux-x86/bin/simg2img as well. ↩︎
You also use udisksctl to mount the image: udisksctl loop-setup -r -f swbins-unsparse.img ↩︎
Android’s metalava takes up enormous amounts of RAM. It attempts to verify the whole Android API at once and is badly designed. ↩︎

SELinux

Sat, 08 Dec 2018 22:15:22 +0200

Sorry this is such a mess, this is more of a braindump…

General

(from SEAndroid-Aalto presentation):

Subjects are primarily processes. Processes that participate in the access control are assigned a domain

Objects are e.g. files, sockets.. Objects that participate in the access control are assigned a type

 X          N        allow   M                 Y
subjects   domains   rules  types          objects
                                 --------------[]
                                /      --------[]
[]---\        /-------------[]---- ---/--------[]
[]    ------[]------/-------[]--- ---/---------[]
[]----------[]\----/--------[]-\----/----------[]
[]      /  -[]-\------------[]--\--------------[]
[]-----/  /   \ \              \/\             []
[]       /     \------------[]--\-\------------[]
[]------/-                       --\-----------[]
                                    -----------[]

An object is always an OS primitive of some sort. This is not a policy issue, this is reality. An object therefore belongs to one or more classes which are predefined by the system proper. A class can be e.g. file, socket, character device, ….

Objects   Classes      Permissions
[]--           /-----------[]
[]--\-------[]-------------[]
[]-/          \------------[]
[]-

A class is associated with a fixed set of permissions (e.g read, write, open..), often closely mapped to system call functionality. Again, the set of available permission is predefined, policy does not change that But! An allow rule includes also the class, and within that the permissions allowed by the rule. Class and permission names are ”well known” as defined by NSA / SELinux

In general: allow [domain] [type] : [class] {[ allowed permissions ]}

[domain]: Subject(a process in a domain)
[type]: Object(a file or other resource)
[class]: Resource class(from predefined set)
[allowed_permissions]: Class-specific permissions that are allowed by this rule

Testing

Install needed tools for the host machine:

apt install setools python-networkx policycoreutils

Testing it out: either mka selinux_policy or make bootimage But to only check if the policy is valid: make -j $(nproc) sepolicy_tests!

Misc

Directories:

system/sepolicy
device/sony/sepolicy
device/sony/<platform>/sepolicy_platform
system/bt/vendor_libs/linux/sepolicy
system/extras/boottime_tools/bootio/sepolicy
…
file_contexts(“real” files, but not necessarily limited to that), genfs_contexts(“generated” files) -> contexts need to exist!
property_contexts -> always need a corresponding type in property.te!

PRODUCT_FULL_TREBLE_OVERRIDE := false
BOARD_USE_ENFORCING_SELINUX := true

Treble and changes from Oreo to Pie binder_in_vendor_violators etc

binder, hwbinder, vndbinder

Cleaning commands:

rm out/target/product/kagura/obj/ETC/*sepolicy* -r
find out/target/product/kagura/ -iname "*sepolicy*" -exec rm -rv {} \;
find out/target/product/kagura/ -iname "*selin*" -exec rm -rv {} \;
find out/target/product/kagura/ -iname "*file_context*" -exec rm -rv {} \;

Tips:

restorecon -R /path
ls -Z /path or ls -Z /path/file
ps -AZ
Always try to find out what the macros mean! Check system/sepolicy/pulic/{te_macros,global_macros} and any file you might find that explains a thing that does not fall within the standard rule_name source_type target_type : class perm_set; schema. Especially binder_use() and similar macros pull in neverallows and things you might not expect.
Have a copy of the te_macros and global_macros printed out!
load_policy to reload policy on device
sesearch and seinfo tools to examine policy(must be installed on host, not available on standard android device)
Check paths with matchpathcon -P <policy file dir> -f <file_contexts_file> /my/path (Important: -P needs to point to the directory containing the policy file, not the policy file itself!)

Examples:

Different app levels:

seapp_contexts
platform_app.te
service_contexts, hwservice_contexts

From domain.te:

Only service_manager_types can be added to service_manager
# TODO: rework this:
neverallow * ~service_manager_type:service_manager { add find };

From init.te:

Init never adds or uses services via service_manager
neverallow init service_manager_type:service_manager { add find };

Or own domain via own myapp.te, with:

type myapp, domain;
type myapp_exec, exec_type, vendor_file_type, file_type;

init_daemon_domain(myapp);
add_service(myapp, qcrilam_service)

typeattribute myapp binder_in_vendor_violators;
binder_use(myapp)

allow myapp_service self:hwservice_manager find;

(for more, see https://stackoverflow.com/questions/28914215/is-it-possible-to-use-the-package-name-as-the-domain-name-in-android-selinux)

Have a look at system/sepolicy/public/te_macros, system/sepolicy/public/global_macros

Google’s guides:

Gotchas

Confusing domains, types vs attributes(?), labels, objects, contexts, …
service_manager (from add_service(domain, service)) vs servicemanager (from binder_use(domain)). Yes, this is extremely confusing!
Confusing public/vendor and private policy folders/“APIs”
Confusing objects(mostly files) and running subjects like “apps”(the Java ones) or “services”(mostly C binaries that run from /system/bin or /vendor/bin, but can also be Java apps OR HAL/HIDL implementations)
Things can be active(searching/finding something) or passive(being labeled something and then being interacted with). Attributes are mostly passive, while subjects and their actions are active.

seapp

Is seapp_contexts a labeling mechanism? Or a sanity check through neverallows? Can you leave out everything apart from name=com.my.app?

seinfo is mainly used to enforce signing requirements, e.g. you have mac_permissions.xml(MAC = Mandatory Access Control) that specifies that anything with seinfo=platform has to be signed by the platform key:

<?xml version="1.0" encoding="utf-8"?><policy>
[...]
<!-- Platform dev key in AOSP -->
<signer signature="@PLATFORM" >
  <seinfo value="platform" />
</signer>
</policy>

Taming a service: QcRilAm

THIS IS OUT OF DATE!

QcRilAm is our own little app/service that proxies requests from rild to android.hardware.radio.am. It was written by oshmoun and is needed to get in-call audio to work. It consists of two parts: One HIDL implementation and one Java app that acts as a service. The two talk via binder.

# Establish vnd_qcril_audio_hwservice
# hwservice_contexts
vendor.qti.hardware.radio.am::IQcRilAudio           u:object_r:vnd_qcril_audio_hwservice:s0
vendor.qti.hardware.radio.am::IQcRilAudioCallback   u:object_r:vnd_qcril_audio_hwservice:s0

The HIDL service(called vendor.qti.hardware.radio.am) gets accessed by rild via hwbinder calls(since it is a hardware service).

# Allow rild to access vendor.qti.hardware.radio.am HIDL services
# rild.te
hwbinder_use(rild)
add_hwservice(rild, vnd_qcril_audio_hwservice)

The persistent android service part of QcRilAm is a java app that will also appear in a user’s app listing called com.sony.qcrilam. It only serves to run its one service called com.sony.qcrilam.QcRilAmService.

Right now, it appears in the process list as a platform_app:

$ ps -AZ | grep qcrilam
u:r:platform_app:s0:c512,c768  u0_a62        2045   585 3685212  80452 SyS_epoll_wait      0 S com.sony.qcrilam

To use a more restricted and thus safer domain:

# Assign com.sony.qcrilam.QcRilAmService a domain
# service_contexts
com.sony.qcrilam.QcRilAmService         u:object_r:qcrilam_service:s0

But wait! We need to define what a qcrilam_service is first!

# service.te
type qcrilam_service, service_manager_type;

Same goes for vnd_qcril_audio_hwservice:

# hwservice.te
type vnd_qcril_audio_hwservice,    hwservice_manager_type;

Alright, now we need to assign qcrilam_service to a domain. We could borrow from an already existing one, but AOSP doesn’t quite provide one that suits us. So we create our own:

# qcrilam.te
type qcrilam, domain;
add_service(qcrilam, qcrilam_service)
# the add_service() macro is equivalent to this:
#allow qcrilam qcrilam_service:service_manager { add find };
# TODO: Not needed with vnd change!
#typeattribute qcrilam binder_in_vendor_violators;
#binder_use(qcrilam_service)
vndbinder_use(qcrilam_service)
binder_call(qcrilam, vnd_qcril_audio_hwservice)

com.sony.qcrilam got assigned the platform_app label by SELinux automatically. This is a very high privilege level, as opposed to untrusted_app contexts. We want qcrilam to find its HIDL cousin via hwservicemanager in addition to being able to call them via binder:

# platform_app.te
allow platform_app vnd_qcril_audio_hwservice:hwservice_manager find;

A final thing to keep in mind is that rild is very greedy with permissions. Modems are a black box we have to contend with. Allow platform apps to talk to rild via binder and vice versa:

# rild.te
binder_call(platform_app, rild);
binder_call(rild, platform_app);

Wrapping up

Something to keep in mind is: You are only allowing or disallowing things, not influencing the flow of programs¹. Even if te macros can look like little function statements, they only dictate what gets allowed, not how a program behaves.

Ok that one is not 100% correct, some programs will try to do things and when they get denied switch to a different strategy. But in general, you will not incluence control flow much with sepolicy. ↩︎

DRAFT: Kernel Compilation

Sat, 20 Jun 2020 21:24:31 +0200

Sony link: How to build and flash a Linux Kernel for AOSP

Unlock bootloader

See Unlock Bootloader on sony.com

Download cross compiler

Linaro or clang

Clang:

git clone https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86 -b android-10.0.0_r39 clang-linux-x86

Linaro GCC: Download

gcc-arm-9.2-2019.12-mingw-w64-i686-arm-none-linux-gnueabihf.tar.xz and gcc-arm-9.2-2019.12-mingw-w64-i686-aarch64-none-linux-gnu.tar.xz from the ARM website, extract the files.

Export cross compiler

Linaro GCC:

export CROSS_COMPILE=<android-dir>/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/aarch64-linux-android-
export CROSS_COMPILE_ARM32=<android-dir>/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/aarch64-linux-android-

For clang:

export CC=clang-linux-x86/clang-r353983c1/bin
export CLANG_TRIPLE=aarch64-linux-gnu

Download kernel source

git clone https://github.com/sonyxperiadev/kernel
cd kernel

Also techpack/audio, data-kernel, wifi modules

Identify defconfig

The defconfig name is constructed using this schema: aosp_<platform>_<codename>_defconfig.

As for the codename, search the Sony Devices overview for the device Code name, e.g. kagura for the Xperia XZ.

For platform, use the “board” name, e.g. tone for Xperia XZ.

So, Xperia XZ becomes aosp_tone_kagura_defconfig.

For easier reference, here’s an overview:

Device name	defconfig	DTBO
Xperia X	`aosp_loire_suzu_defconfig`	false
Xperia X Compact	`aosp_loire_kugo_defconfig`	false
Xperia Touch	`aosp_loire_blanc_defconfig`	false
Xperia X Performance	`aosp_tone_dora_defconfig`	false
Xperia XZ	`aosp_tone_kagura_defconfig`	false
Xperia XZs	`aosp_tone_keyaki_defconfig`	false
Xperia XZ Premium	`aosp_yoshino_maple_defconfig`	false
Xperia XZ1	`aosp_yoshino_poplar_defconfig`	false
Xperia XZ1 Compact	`aosp_yoshino_lilac_defconfig`	false
Xperia XA2	`aosp_nile_pioneer_defconfig`	false
Xperia XA2 Ultra	`aosp_nile_discovery_defconfig`	false
Xperia XA2 Plus	`aosp_nile_voyager_defconfig`	false
Xperia XZ2	`aosp_tama_akari_defconfig`	true
Xperia XZ2 Compact	`aosp_tama_apollo_defconfig`	true
Xperia XZ3	`aosp_tama_akatsuki_defconfig`	true
Xperia 10	`aosp_ganges_kirin_defconfig`	true
Xperia 10 Plus	`aosp_ganges_mermaid_defconfig`	true
Xperia 1	`aosp_kumano_griffin_defconfig`	true
Xperia 5	`aosp_kumano_bahamut_defconfig`	true
Xperia 10 II	`aosp_seine_pdx201_defconfig`	true

Configure kernel

make ARCH=arm64 CROSS_COMPILE=$CROSS_COMPILE CROSS_COMPILE_ARM32=$CROSS_COMPILE_ARM32 <device_defconfig>

For clang:

make ARCH=arm64 CC=$CC CLANG_TRIPLE=$CLANG_TRIPLE CROSS_COMPILE=$CROSS_COMPILE CROSS_COMPILE_ARM32=$CROSS_COMPILE_ARM32 <device_defconfig>

Build kernel

make \
  ARCH=arm64 \
  CROSS_COMPILE=$CROSS_COMPILE \
  CROSS_COMPILE_ARM32=$CROSS_COMPILE_ARM32 \
  -j $(nproc)

For clang:

make \
  ARCH=arm64 \
  CC=$CC \
  CLANG_TRIPLE=$CLANG_TRIPLE \
  CROSS_COMPILE=$CROSS_COMPILE \
  CROSS_COMPILE_ARM32=$CROSS_COMPILE_ARM32 \
  -j $(nproc)

Copy kernel and dtbo

Devices with appended DTB

Copy <kernel>/arch/arm64/boot/Image.gz-dtb to <android-dir>/kernel/sony/msm-4.14/common-kernel/kernel-dtb-<device>.

DTBO devices

Copy <kernel>/arch/arm64/boot/Image.gz to <android-dir>/kernel/sony/msm-4.14/common-kernel/kernel-dtb-<device>.

Then, build mkdtimg from inside your Android tree and run it:

export MKDTIMG=<android-dir>/out/host/linux-x86/bin/mkdtimg
$MKDTIMG create <android-dir>kernel/sony/msm-4.14/common-kernel/dtbo-<device>.img \
  `find <kernel-dir>/arch/arm64/boot/dts -name "*.dtbo"`

Build bootimage and dtboimage

Navigate into top-level Android directory.

source build/envsetup.sh && lunch
make -j $(nproc) bootimage

The DTBO image will be generated automatically when building boot image.

Download fastboot

On Linux: Install fastboot package

On Windows: Link to drivers

Flash via fastboot

Make sure the current slot is correct:

$ fastboot getvar current-slot
-> "a"

Else, set it:

fastboot set_active a

Then, flash images:

fastboot flash boot out/target/product/<device>/boot.img
# For DTBO devices:
fastboot flash dtbo out/target/product/<device>/dtbo.img

Finally:

fastboot reboot

# https://t.me/PixelExperienceSony/34425
#!/bin/bash

export ANDROID_ROOT=/home/kgvarunkanth/kgv/arrow
export KERNEL_TOP=$ANDROID_ROOT/kernel/sony/msm-4.14
export KERNEL_TMP=$ANDROID_ROOT/out/kernel-4.14
export CC=$ANDROID_ROOT/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/
export CC32=$ANDROID_ROOT/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin/
export CLANG=$ANDROID_ROOT/prebuilts/clang/host/linux-x86/clang-r353983c/bin/
export PATH=$CC:$CC32:$CLANG:$PATH
rm -rf $KERNEL_TMP
make O=$KERNEL_TMP clean
make O=$KERNEL_TMP mrproper
make O=$KERNEL_TMP ARCH=arm64 aosp_yoshino_maple_defconfig
make O=$KERNEL_TMP ARCH=arm64 CC=clang CLANG_TRIPLE=aarch64-linux-gnu- CROSS_COMPILE=aarch64-linux-android- CROSS_COMPILE_ARM32=arm-linux-androideabi- -j1

Android Init Stages

Mon, 25 Feb 2019 16:49:06 +0100

Stages

Init stage	Contents
`early-init`	Very first stage of initialization. Used for SELinux and OOM settings
`init`	Creates file systems, mount points, and writes kernel variables
`early-fs`	Run just before filesystems are ready to be mounted
`fs`	Specifies which partitions to load
`post-fs`	Commands to run after filesystems (other than `/data`) are mounted
`post-fs-data`	`/data` decrypted (if necessary) and mounted
`early-boot`	Run after property service has been initialized, but before booting rest
`boot`	Normal boot commands
`charger`	Commands used when device is in charger mode

Adapted from Jonathan Levin’s excellent Android Internals: A Confectioner’s Cookbook, chapter 4.

Classes

Programs can be started based on init stage triggers:

service my_service /bin/mysvc
    user system
    group system
    disabled

on boot
    start my_service

But it is more common to assign a class to a service, based on which it is started:

service my_service /bin/mysvc
    class core
    user system
    group system

The service will be started based on class_start core which happens on boot.

TODO: Overview of classes

Class	Comment
`core`	Never shut down after restarting
`main`	Shut down and then restart after the disk password is entered
`late_start`	Does not start until after /data has been decrypted and mounted

Trigger	Classes
`on charger`	`charger`
`on late-fs`	`early-hal`
`on nonencrypted`	`hal`, `core`
`on boot`	`hal`, `core`

Keywords (selection)

Keyword	Action
`start`	Starts a service, even when it’s disabled
`class core`	Makes a service part of a group.
`class_start`	Starts a group. Once this group is started, the service is started only if it is enabled (services can be disabled by having `disabled` in the service spec, or `disable <service>`)
`enable`	Allows a service to be started as part of a `class_start`. If the class is already started, this’ll schedule the service to be started
`disable`	Prevent a service from being started as part of `class_start`. This does not stop the service if it’s already running

Changes with encryption

With the introduction of system-wide encryption into Android, the init process gains a few more tricks:

vold.decrypt triggers
class early_hal for crucial services needed for decryption, e.g. keymaster and gatekeeper (TODO: Add link to AOSP init.rc for class_start)
restarting framework, reboot, minimal UI stuff
nonencrypted trigger, which, despite its name, always gets triggered and which pulls in class_start main and class_start late_start

on property:vold.decrypt=trigger_restart_min_framework
    class_start main

on property:vold.decrypt=trigger_restart_framework
    stop surfaceflinger
    start surfaceflinger
    class_start main
    class_start late_start

on property:vold.decrypt=trigger_shutdown_framework
    class_reset late_start
    class_reset main

Gone in master/Q: Stop using trigger_reset_main

Sony Devices

Sun, 10 Feb 2019 21:21:51 +0100

Snapdragon 865 devices (edo board, 2020)

Internal SoC codename: sm8250

Platform: edo

Latest Open Devices Kernel: 4.19

Device name	Code name	Product (SS, DS)
Xperia 1 II	pdx203	xqat51, xqat52
Xperia 5 II	pdx206	xqas52

Peculiarities: Dynamic Android Partitions, A/B, has DTBO partition.