mirror/docker-lineage-cicd
1
0
Fork 0
mirror of https://github.com/lineageos4microg/docker-lineage-cicd synced 2024-11-09 10:09:56 +01:00
Code Issues

Add multibranch support

Browse Source 
It's now possible to build multiple branches with the same docker image.
A full LineageOS mirror is downloaded and created locally (~200 GB).
This commit is contained in:
Nicola Corna 2017-11-10 00:26:56 +01:00
parent 4176346648
commit 324a10e762
4 changed files with 369 additions and 204 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
Dockerfile
View File

@ -1,10 +1,11 @@
FROM ubuntu:16.04
FROM debian:stretch
MAINTAINER Nicola Corna <nicola@corna.info>
# Environment variables
#######################
ENV SRC_DIR /srv/src
ENV TMP_DIR /srv/tmp
ENV CCACHE_DIR /srv/ccache
ENV ZIP_DIR /srv/zips
ENV LMANIFEST_DIR /srv/local_manifests
@ -28,7 +29,7 @@ ENV USE_CCACHE 1
# for no limit.
ENV CCACHE_SIZE 50G
# Environment for the LineageOS Branch name
# Environment for the LineageOS branches name
# See https://github.com/LineageOS/android_vendor_cm/branches for possible options
ENV BRANCH_NAME 'cm-14.1'
@ -52,6 +53,10 @@ ENV OTA_URL ''
ENV USER_NAME 'LineageOS Buildbot'
ENV USER_MAIL 'lineageos-buildbot@docker.host'
# Include proprietary files, downloaded automatically from github.com/TheMuppets/
# Only some branches are supported
ENV INCLUDE_PROPRIETARY true
# If you want to start always fresh (re-download all the source code everytime) set this to 'true'
ENV CLEAN_SRCDIR false
@ -125,6 +130,7 @@ ENV OPENDELTA_BUILDS_JSON ''
# Create Volume entry points
############################
VOLUME $SRC_DIR
VOLUME $TMP_DIR
VOLUME $CCACHE_DIR
VOLUME $ZIP_DIR
VOLUME $LMANIFEST_DIR
@ -140,6 +146,7 @@ COPY src/ /root/
# Create missing directories
############################
RUN mkdir -p $SRC_DIR
RUN mkdir -p $TMP_DIR
RUN mkdir -p $CCACHE_DIR
RUN mkdir -p $ZIP_DIR
RUN mkdir -p $LMANIFEST_DIR
@ -150,15 +157,18 @@ RUN mkdir -p $USERSCRIPTS_DIR
# Install build dependencies
############################
RUN echo 'deb http://deb.debian.org/debian sid main' >> /etc/apt/sources.list
RUN echo 'deb http://deb.debian.org/debian experimental main' >> /etc/apt/sources.list
COPY apt_preferences /etc/apt/preferences
RUN apt-get -qq update
RUN apt-get -qqy upgrade
RUN apt-get install -y bc bison build-essential ccache cron curl flex \
g++-multilib gcc-multilib git gnupg gperf imagemagick lib32ncurses5-dev \
lib32readline6-dev lib32z1-dev libesd0-dev liblz4-tool libncurses5-dev \
lib32readline-dev lib32z1-dev libesd0-dev liblz4-tool libncurses5-dev \
libsdl1.2-dev libssl-dev libwxgtk3.0-dev libxml2 libxml2-utils lzop \
maven openjdk-8-jdk pngcrush rsync schedtool squashfs-tools wget xdelta3 \
xsltproc zip zlib1g-dev
maven openjdk-7-jdk openjdk-8-jdk pngcrush procps python rsync schedtool \
squashfs-tools wget xdelta3 xsltproc yasm zip zlib1g-dev
RUN curl https://storage.googleapis.com/git-repo-downloads/repo > /usr/local/bin/repo
RUN chmod a+x /usr/local/bin/repo

60
README.md
View File

@ -2,23 +2,17 @@
Docker microservice for LineageOS Continuous Integration and Continous Deployment
## Why
Because I always believe that even advanced technologies should be available to everyone. This is a tentative to offer everyone the possibility to build his own images of LineageOS, when he wants, how he wants. You don't have to wait anymore for build bots. No more scene drama. Just build and enjoy your favourite Android ROM.
## Why Docker?
Because I'm a big fan of isolating everything if possible. I don't want to reinstall my OS or triage with dirty packages, just because today I need somethng, and tomorrow I'll need something else.
This branch contains a version suitable to build for multiple devices on different branches with the same Docker image: to do so it is necessary to download the full LineageOS mirror, about 200 GB. You probably want the more simple version in the master branch.
## Requirements
- At least Dual Core CPU (Higher is better)
- At least 6GB RAM (Higher is better)
- At least 250GB HDD Space (Higher is better)
- At least 500GB HDD Space (Higher is better)
### Android propretary binaries
By default when you build Android from scratch you need to pull the Binaries of your interested device via ADB. Although via this Docker is not possible to do so (would imply having all the devices connected to that machine and ideally know how to switch from one to the other before pulling). Therefore, I highly suggest to download this manifest (https://github.com/TheMuppets/manifests) inside your mapped `/srv/local_manifests` folder.
By default when you build Android from scratch you need to pull the binaries of your interested device via ADB. When the INCLUDE_PROPRIETARY variable is set to true (default), the proprietary binaries are downloaded automatically from [TheMuppets repository](https://github.com/TheMuppets); if you want to provide your own binaries, set this variable to false and add a corresponding XML in the local_manifests volume.
## How it works
@ -55,6 +49,7 @@ Instead of scheduling the execution, build it now and exit.
For each device, create a subdir `device_codename` in /home/user/zips and move the builds there.
```
docker run \
--cap-add=SYS_ADMIN \
-d \
-e "USER_NAME=John Doe" \
-e "USER_MAIL=john.doe@awesome.email" \
@ -74,6 +69,7 @@ Finally provide a custom OTA URL for this ROM so users can update using built-in
```
docker run \
--restart=always \
--cap-add=SYS_ADMIN \
-d \
-e "USER_NAME=John Doe" \
-e "USER_MAIL=john.doe@awesome.email" \
@ -90,17 +86,19 @@ docker run \
```
### Custom mode
You can also apply some modifications to the LineageOS code before building it. This example is the build script used for [LineageOS for microG](https://lineage.microg.org), which has integrated microG apps and F-Droid (with F-Droid Privileged Extension).
You can also apply some modifications to the LineageOS code before building it.
```
docker run \
--name=lineage-$(date +%Y%m%d_%H%M) \
--cap-add=SYS_ADMIN \
-d \
-e "USER_NAME=John Doe" \
-e "USER_MAIL=john.doe@awesome.email" \
-e "WITH_SU=false" \
-e "RELEASE_TYPE=microG" \
-e "RELEASE_TYPE=myrelease" \
-e "BRANCH_NAME=cm-14.1" \
-e "DEVICE_LIST=$DEVICES" \
-e "OTA_URL=https://api.lineage.microg.org" \
-e "OTA_URL=http://cool.domain/api" \
-e "CRONTAB_TIME=now" \
-e "SIGNATURE_SPOOFING=restricted" \
-e "CUSTOM_PACKAGES=GmsCore GsfProxy FakeStore FDroid FDroidPrivilegedExtension MozillaNlpBackend NominatimNlpBackend com.google.android.maps.jar" \
@ -120,6 +118,44 @@ docker run \
-v "/home/user/lineage_scripts:/srv/userscripts" \
lineageos4microg/docker-lineage-cicd
```
### Multiple devices, multiple branches
Specify the branches and the devices in the corresponding variables, separated by a comma.
This example is the build script used for [LineageOS for microG](https://lineage.microg.org), which has integrated microG apps and F-Droid (with F-Droid Privileged Extension).
```
docker run \
--name=lineage-$(date +%Y%m%d_%H%M) \
--cap-add=SYS_ADMIN \
-d \
-e "USER_NAME=John Doe" \
-e "USER_MAIL=john.doe@awesome.email" \
-e "WITH_SU=false" \
-e "INCLUDE_PROPRIETARY=true" \
-e "RELEASE_TYPE=microG" \
-e "BRANCH_NAME=cm-13.0,cm-14.1" \
-e "DEVICE_LIST_CM_13_0=$DEVICES_CM13_0" \
-e "DEVICE_LIST_CM_14_1=$DEVICES_CM14_1" \
-e "OTA_URL=https://api.lineage.microg.org" \
-e "CRONTAB_TIME=now" \
-e "SIGNATURE_SPOOFING=restricted" \
-e "CUSTOM_PACKAGES=GmsCore GsfProxy FakeStore FDroid FDroidPrivilegedExtension MozillaNlpBackend NominatimNlpBackend com.google.android.maps.jar" \
-e "SIGN_BUILDS=true" \
-e "CLEAN_OUTDIR=false" \
-e "CLEAN_AFTER_BUILD=true" \
-e "ZIP_SUBDIR=true" \
-e "LOGS_SUBDIR=true" \
-e "DELETE_OLD_ZIPS=3" \
-e "DELETE_OLD_LOGS=3" \
-e "CCACHE_SIZE=540G" \
-v "/home/user/cache:/srv/ccache" \
-v "/home/user/lineage:/srv/src" \
-v "/home/user/zips:/srv/zips" \
-v "/home/user/lineage_manifests:/srv/local_manifests" \
-v "/home/user/lineage_keys:/srv/keys" \
-v "/home/user/logs:/srv/logs" \
-v "/home/user/tmp:/srv/tmp" \
lineageos4microg/docker-lineage-cicd:multibranch
```
with the following XML in local_manifests
```
<?xml version="1.0" encoding="UTF-8"?>

28
apt_preferences Normal file
View File

@ -0,0 +1,28 @@
Package: *
Pin: release n=experimental
Pin-Priority: -1
Package: *
Pin: release n=sid
Pin-Priority: -1
Package: openjdk-7-jdk
Pin: release n=experimental
Pin-Priority: 500
Package: openjdk-7-jre
Pin: release n=experimental
Pin-Priority: 500
Package: openjdk-7-jre-headless
Pin: release n=experimental
Pin-Priority: 500
Package: libfontconfig1
Pin: release n=sid
Pin-Priority: 500
Package: fontconfig-config
Pin: release n=sid
Pin-Priority: 500

179
src/build.sh
View File

@ -17,7 +17,8 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
if ! [ -z "$DEVICE_LIST" ]; then
IFS=','
shopt -s dotglob
# cd to working directory
cd $SRC_DIR
@ -27,41 +28,110 @@ if ! [ -z "$DEVICE_LIST" ]; then
/root/userscripts/begin.sh
fi
# If the source directory is empty
if ! [ "$(ls -A $SRC_DIR)" ]; then
# Initialize repository
echo ">> [$(date)] Initializing repository"
yes | repo init -u https://github.com/lineageos/android.git -b ${BRANCH_NAME:-cm-14.1}
# If requested, clean the OUT dir in order to avoid clutter
if [ "$CLEAN_OUTDIR" = true ]; then
echo ">> [$(date)] Cleaning '$ZIP_DIR'"
rm $ZIP_DIR/*
fi
# Treat DEVICE_LIST as DEVICE_LIST_<first_branch>
first_branch=$(cut -d ',' -f 1 <<< $BRANCH_NAME)
if [ ! -z "$DEVICE_LIST" ]; then
device_list_first_branch="DEVICE_LIST_$(sed 's/[^[:alnum:]]/_/g' <<< $first_branch)"
device_list_first_branch=${device_list_first_branch^^}
read $device_list_first_branch <<< "$DEVICE_LIST,${!device_list_first_branch}"
fi
# If needed, migrate from the old SRC_DIR structure
if [ -d $SRC_DIR/.repo ]; then
echo ">> [$(date)] Removing old repository"
rm -rf $SRC_DIR/*
fi
mkdir -p $TMP_DIR/device
mkdir -p $TMP_DIR/workdir
mkdir -p $TMP_DIR/merged
mkdir -p $SRC_DIR/mirror
cd $SRC_DIR/mirror
if [ ! -d .repo ]; then
echo ">> [$(date)] Initializing mirror repository"
yes | repo init -q -u https://github.com/LineageOS/mirror --mirror --no-clone-bundle -p linux
fi
# Copy local manifests to the appropriate folder in order take them into consideration
echo ">> [$(date)] Copying '$LMANIFEST_DIR/*.xml' to '$SRC_DIR/.repo/local_manifests/'"
rsync -a --delete --exclude 'roomservice.xml' --include '*.xml' --exclude '*' $LMANIFEST_DIR/ $SRC_DIR/.repo/local_manifests/
echo ">> [$(date)] Copying '$LMANIFEST_DIR/*.xml' to 'mirror/.repo/local_manifests/'"
mkdir -p .repo/local_manifests
rsync -a --delete --exclude 'roomservice.xml' --include '*.xml' --exclude '*' $LMANIFEST_DIR .repo/local_manifests/
if [ "$INCLUDE_PROPRIETARY" = true ]; then
wget -q -O .repo/local_manifests/proprietary.xml "https://raw.githubusercontent.com/TheMuppets/manifests/mirror/default.xml"
fi
echo ">> [$(date)] Syncing mirror repository"
repo sync -q --no-clone-bundle
for branch in $BRANCH_NAME; do
branch_dir=$(sed 's/[^[:alnum:]]/_/g' <<< $branch)
branch_dir=${branch_dir^^}
device_list_cur_branch="DEVICE_LIST_$branch_dir"
if [ ! -z "$branch" ] && [ ! -z "${!device_list_cur_branch}" ]; then
mkdir -p $SRC_DIR/$branch_dir
cd $SRC_DIR/$branch_dir
echo ">> [$(date)] Branch: $branch"
echo ">> [$(date)] Devices: ${!device_list_cur_branch}"
# Reset the current git status of "vendor/cm" (remove previous changes) if the directory exists
if [ -d "vendor/cm" ]; then
cd vendor/cm
git reset -q --hard
cd $SRC_DIR
cd ../..
fi
# Reset the current git status of "frameworks/base" (remove previous changes) if the directory exists
if [ -d "frameworks/base" ]; then
cd frameworks/base
git reset -q --hard
cd $SRC_DIR
cd ../..
fi
# Sync the source code
echo ">> [$(date)] Syncing repository"
builddate=$(date +%Y%m%d)
repo sync -q
if [ ! -d .repo ]; then
echo ">> [$(date)] Initializing branch repository"
yes | repo init -q -u https://github.com/LineageOS/android.git --reference $SRC_DIR/mirror/ -b $branch
fi
# If needed, apply the MicroG's signature spoofing patch
# Copy local manifests to the appropriate folder in order take them into consideration
echo ">> [$(date)] Copying '$LMANIFEST_DIR/*.xml' to '.repo/local_manifests/'"
mkdir -p .repo/local_manifests
rsync -a --delete --exclude 'roomservice.xml' --include '*.xml' --exclude '*' $LMANIFEST_DIR .repo/local_manifests/
if [ "$INCLUDE_PROPRIETARY" = true ]; then
if [[ $branch =~ .*cm\-13\.0.* ]]; then
themuppets_branch=cm-13.0
elif [[ $branch =~ .*cm-14\.1.* ]]; then
themuppets_branch=cm-14.1
else
themuppets_branch=cm-14.1
echo ">> [$(date)] Can't find a matching branch on github.com/TheMuppets, using $themuppets_branch"
fi
wget -q -O .repo/local_manifests/proprietary.xml "https://raw.githubusercontent.com/TheMuppets/manifests/$themuppets_branch/muppets.xml"
fi
echo ">> [$(date)] Syncing branch repository"
builddate=$(date +%Y%m%d)
repo sync -q -c
android_version=$(sed -n -e 's/^\s*PLATFORM_VERSION := //p' build/core/version_defaults.mk)
android_version_major=$(cut -d '.' -f 1 <<< $android_version)
# If needed, apply the microG's signature spoofing patch
if [ "$SIGNATURE_SPOOFING" = "yes" ] || [ "$SIGNATURE_SPOOFING" = "restricted" ]; then
# Determine which patch should be applied to the current Android source tree
patch_name=""
android_version=$(sed -n -e 's/^\s*PLATFORM_VERSION := //p' build/core/version_defaults.mk)
case $android_version in
4.4* ) patch_name="android_frameworks_base-KK-LP.patch" ;;
5.* ) patch_name="android_frameworks_base-KK-LP.patch" ;;
@ -80,24 +150,12 @@ if ! [ -z "$DEVICE_LIST" ]; then
sed 's/android:protectionLevel="dangerous"/android:protectionLevel="signature|privileged"/' "/root/signature_spoofing_patches/$patch_name" | patch --quiet -p1
fi
git clean -q -f
cd ../..
else
echo ">> [$(date)] ERROR: can't find a suitable signature spoofing patch for the current Android version ($android_version)"
exit 1
fi
fi
cd $SRC_DIR
# If requested, clean the OUT dir in order to avoid clutter
if [ "$CLEAN_OUTDIR" = true ]; then
echo ">> [$(date)] Cleaning '$ZIP_DIR'"
cd $ZIP_DIR
rm *
cd $SRC_DIR
fi
# Prepare the environment
echo ">> [$(date)] Preparing build environment"
source build/envsetup.sh > /dev/null
echo ">> [$(date)] Setting \"$RELEASE_TYPE\" as release type"
sed -i '/#.*Filter out random types/d' vendor/cm/config/common.mk
@ -120,23 +178,45 @@ if ! [ -z "$DEVICE_LIST" ]; then
sed -i "1s;^;PRODUCT_DEFAULT_DEV_CERTIFICATE := $KEYS_DIR/releasekey\nPRODUCT_OTA_PUBLIC_KEYS := $KEYS_DIR/releasekey\nPRODUCT_EXTRA_RECOVERY_KEYS := $KEYS_DIR/releasekey\n\n;" vendor/cm/config/common.mk
fi
if [ "$android_version_major" -ge "7" ]; then
jdk_version=8
elif [ "$android_version_major" -ge "5" ]; then
jdk_version=7
else
echo ">> [$(date)] ERROR: $branch requires a JDK version too old (< 7); aborting"
exit 1
fi
echo ">> [$(date)] Using OpenJDK $jdk_version"
update-java-alternatives -s java-1.$jdk_version.0-openjdk-amd64 > /dev/null 2>&1
# Prepare the environment
echo ">> [$(date)] Preparing build environment"
source build/envsetup.sh > /dev/null
if [ -f /root/userscripts/before.sh ]; then
echo ">> [$(date)] Running before.sh"
/root/userscripts/before.sh
fi
# Cycle DEVICE_LIST environment variable, to know which one may be executed next
IFS=','
for codename in $DEVICE_LIST; do
for codename in ${!device_list_cur_branch}; do
if ! [ -z "$codename" ]; then
currentdate=$(date +%Y%m%d)
if [ "$builddate" != "$currentdate" ]; then
# Sync the source code
echo ">> [$(date)] Syncing repository"
echo ">> [$(date)] Syncing mirror repository"
builddate=$currentdate
repo sync -q
cd $SRC_DIR/mirror
repo sync -q --no-clone-bundle
echo ">> [$(date)] Syncing branch repository"
cd $TMP_DIR/$branch_dir
repo sync -q -c
fi
if ! [ -z "$codename" ]; then
mount -t overlay overlay -o lowerdir=$SRC_DIR/$branch_dir,upperdir=$TMP_DIR/device,workdir=$TMP_DIR/workdir $TMP_DIR/merged
cd $TMP_DIR/merged
if [ "$ZIP_SUBDIR" = true ]; then
zipsubdir=$codename
mkdir -p $ZIP_DIR/$zipsubdir
@ -159,7 +239,7 @@ if ! [ -z "$DEVICE_LIST" ]; then
fi
# Start the build
echo ">> [$(date)] Starting build for $codename" | tee -a $DEBUG_LOG
echo ">> [$(date)] Starting build for $codename, $branch branch" | tee -a $DEBUG_LOG
if brunch $codename >> $DEBUG_LOG 2>&1; then
currentdate=$(date +%Y%m%d)
if [ "$builddate" != "$currentdate" ]; then
@ -167,7 +247,7 @@ if ! [ -z "$DEVICE_LIST" ]; then
fi
if [ "$BUILD_DELTA" = true ]; then
if [ -d "$SRC_DIR/delta_last/$codename/" ]; then
if [ -d "delta_last/$codename/" ]; then
# If not the first build, create delta files
echo ">> [$(date)] Generating delta files for $codename" | tee -a $DEBUG_LOG
cd /root/delta
@ -179,24 +259,26 @@ if ! [ -z "$DEVICE_LIST" ]; then
if [ "$DELETE_OLD_DELTAS" -gt "0" ]; then
/usr/bin/python /root/clean_up.py -n $DELETE_OLD_DELTAS $DELTA_DIR >> $DEBUG_LOG 2>&1
fi
cd $TMP_DIR/merged
else
# If the first build, copy the current full zip in $SRC_DIR/delta_last/$codename/
# If the first build, copy the current full zip in $SRC_DIR/merged/delta_last/$codename/
echo ">> [$(date)] No previous build for $codename; using current build as base for the next delta" | tee -a $DEBUG_LOG
mkdir -p $SRC_DIR/delta_last/$codename/ >> $DEBUG_LOG 2>&1
find out/target/product/$codename -name 'lineage-*.zip' -type f -maxdepth 1 -exec cp {} $SRC_DIR/delta_last/$codename/ \; >> $DEBUG_LOG 2>&1
mkdir -p delta_last/$codename/ >> $DEBUG_LOG 2>&1
find out/target/product/$codename -name 'lineage-*.zip' -type f -maxdepth 1 -exec cp {} $SRC_DIR/merged/delta_last/$codename/ \; >> $DEBUG_LOG 2>&1
fi
fi
# Move produced ZIP files to the main OUT directory
echo ">> [$(date)] Moving build artifacts for $codename to '$ZIP_DIR/$zipsubdir'" | tee -a $DEBUG_LOG
cd $SRC_DIR/out/target/product/$codename
cd out/target/product/$codename
for build in lineage-*.zip; do
sha256sum $build > $ZIP_DIR/$zipsubdir/$build.sha256sum
done
find . -name 'lineage-*.zip*' -type f -maxdepth 1 -exec mv {} $ZIP_DIR/$zipsubdir/ \; >> $DEBUG_LOG 2>&1
cd $SRC_DIR
cd $TMP_DIR/merged
else
echo ">> [$(date)] Failed build for $codename" | tee -a $DEBUG_LOG
fi
# Remove old zips and logs
if [ "$DELETE_OLD_ZIPS" -gt "0" ]; then
/usr/bin/python /root/clean_up.py -n $DELETE_OLD_ZIPS $ZIP_DIR
@ -207,13 +289,22 @@ if ! [ -z "$DEVICE_LIST" ]; then
# Clean everything, in order to start fresh on next build
if [ "$CLEAN_AFTER_BUILD" = true ]; then
echo ">> [$(date)] Cleaning build for $codename" | tee -a $DEBUG_LOG
rm -rf $SRC_DIR/out/target/product/$codename/ >> $DEBUG_LOG 2>&1
rm -rf $SRC_DIR/merged/out/target/product/$codename/ >> $DEBUG_LOG 2>&1
fi
if [ -f /root/userscripts/post-build.sh ]; then
echo ">> [$(date)] Running post-build.sh for $codename" >> $DEBUG_LOG 2>&1
/root/userscripts/post-build.sh $codename >> $DEBUG_LOG 2>&1
fi
echo ">> [$(date)] Finishing build for $codename" | tee -a $DEBUG_LOG
cd $TMP_DIR
umount $TMP_DIR/merged
echo ">> [$(date)] Cleaning source dir for device $codename"
rm -rf device/*
fi
done
fi
done
@ -235,4 +326,4 @@ if ! [ -z "$DEVICE_LIST" ]; then
echo ">> [$(date)] Running end.sh"
/root/userscripts/end.sh
fi
fi