Michael Pöhn
3187d2cbcf
revised build server creation
2017-05-23 20:06:06 +02:00
Michael Pöhn
8abd3f1cbc
auto-lookup vm provider based on available executables; more fault tolerant vagrant package
2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
bba6b8ab0a
fixed reading libvirt box image size
2017-05-23 20:06:06 +02:00
Michael Pöhn
1bd51966b8
vmtools debug logging for check_output calls
2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
5dbcd0e9bd
added box handling to vmtools
2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
01b6473823
refactored kvm_package to vmtools
2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
5580a685db
added makebuildserver option for keeping vagrant box
...
This is very useful for debugging this process, and also for people
who might want to keep a working copy of the box.
2017-05-23 20:06:06 +02:00
Michael Pöhn
d180aa2658
fix virsh destroy parameters
2017-05-23 20:06:06 +02:00
Michael Pöhn
4347c10d9d
use configured vm provider in when calling destroy in makebuildserver
2017-05-23 20:06:06 +02:00
Michael Pöhn
a414aa00ff
use overhauled mv destroy code in build.py
2017-05-23 20:06:06 +02:00
Michael Pöhn
92fada803e
overhauled and moved destroying builder vm to vmtools.py
2017-05-23 20:04:08 +02:00
Michael Pöhn
fb03e17849
fdroid build: added sleep after destroy/undefine
2017-05-23 20:04:08 +02:00
Michael Pöhn
718d01dea2
makebuildserver added sleep after destroy/undefine
2017-05-23 20:04:08 +02:00
Michael Pöhn
3c4b1dec84
makebuildserver more robust codepath for vagrant destroy
2017-05-23 20:04:08 +02:00
Michael Pöhn
7e8f7c65bc
makebuildserver added failsafe when destroy vagrant vm
2017-05-23 20:04:08 +02:00
Michael Pöhn
70a827d59a
makebuildserver use virsh instead of libvirt for forcing domain off
2017-05-23 20:04:08 +02:00
Michael Pöhn
8e5446068b
makebuildserver debug logging when initial provisioning fails
2017-05-23 20:04:08 +02:00
Michael Pöhn
16b609215e
overhauled makebuildserver libvirt vm cleanup
2017-05-23 20:04:08 +02:00
Michael Pöhn
440509cf8a
makebuildserver debugging log details
2017-05-23 20:04:08 +02:00
Michael Pöhn
2aa5523011
makebuildserver prune gloabal vagrant status when purging broken VMs
2017-05-23 20:04:08 +02:00
Michael Pöhn
8cfd3ca770
delete .vagrant dir when cleaning up buildserver vm
2017-05-23 20:04:08 +02:00
Michael Pöhn
437ff7c3f0
jenkins makebuildserver fail if vagrant box was not created
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
2993674aa8
calculate correct size for buildserver-box in makebuildserver
...
https://gitlab.com/fdroid/fdroidserver/issues/238#note_24000153
"Our hard-coded image size meta-data (1000) is for some interpreted as less
than the size of the box-image by my kvm setup. This makes grub/initrd
refuse to boot. So I've changed the metadata size to 9999 which resulted in
an actually booting vm. I can log in on the builder-vm via virt-manager
and virsh.
2017-05-23 20:04:08 +02:00
Michael Pöhn
413c3836d5
auto-reset broken builder vm
2017-05-23 20:04:08 +02:00
Michael Pöhn
ce3c959ce5
scan vm provider again after re-creating builder vm
2017-05-23 20:04:08 +02:00
Michael Pöhn
c54e0565d7
added some debug listings to jenkins makebuildserver script
2017-05-23 20:04:08 +02:00
Michael Pöhn
34cddd3be8
delete associated libvirt domain/image when destroying builder vm
2017-05-23 20:04:08 +02:00
Michael Pöhn
fad98eeb7f
build: fixed kvm snapshot support; makebuildserver: setup kvm ssh credentials
2017-05-23 20:04:08 +02:00
Michael Pöhn
fa2d44ee94
added libvirt vm-provider support to build.py
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
69e4b91d3f
makebuildserver: package up KVM VM as a vagrant box
...
`vagrant package` does not work with KVM, so we have to hack together our
own until someone implements it (suppose we should do it). This is a hacked
up version based on:
https://github.com/vagrant-libvirt/vagrant-libvirt/blob/d7d440ea8f24f698a93a4c5b9a5149acef469579/tools/create_box.sh
#238
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
988ac21e7f
buildserver: make --clean destroy reliably
...
This prevents v.destroy() from running if Vagrantfile.yaml does not exist,
since that is required for vagrant to run: is the core config including the
name of the box, etc. Otherwise, it would exit with an error.
This also does complete cleanup when using libvirt.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
299ed82a88
buildserver: consolidate boxfile export code into one block
...
This is just for clarity, and moving more code into the main() function.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
daade7656a
buildserver: add copy_caches_from_host config option
...
For people using slow, expensive, and/or flaky internet, liberal use of
caching can make a huge difference. The restricted environment of the
gpjenkins box has been a good test environment for this (Tor-only,
whitelist of allowed IPs to visit, home internet connection).
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
4cde71552f
buildserver: run_via_vagrant_ssh() to run cmds via python-vagrant
...
This moves the last vagrant call in a subprocess.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
6464ec55b7
buildserver: move code into main() method to always stop thread
...
By running the whole program in a main() function, it can be wrapped in
try/finally in order to stop the background display thread. This is also
done in ./fdroid, its standard practice for Python CLI utilities.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
675500ad88
buildserver: display verbose logging in a background tail
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
4b03c3d42d
buildserver: replace custom code with python-vagrant
...
I ran into some annoying issues with UTF-8 output in the vagrant logs, and
it was hard to solve. So I switched to using python-vagrant, which handles
it all for us. Its been around since 2012, has a number of contributors,
and is still actively maintained, so it seems like a good bet. I also
packaged it for Debian, including a backport in jessie-backports.
On Debian/jessie, do `apt-get install python3-vagrant/jessie-backports`
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
7ef0d5dfd8
include class like UNIX tail -f for displaying logs
...
This allows fdroidserver to easily log activity while displaying it at the
same time.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
c62e3fd0cc
move bulk of reproducible_fdroid_build_apps.sh to jenkins-build
...
This lets us quickly and frequently test things.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
1710428de1
Merge branch 'exceptions' into 'master'
...
Replace sys.exit() in non-main functions by exceptions
See merge request !273
2017-05-22 20:50:54 +00:00
Torsten Grote
1fcd8e63a3
Replace sys.exit() in non-main functions by exceptions
...
Also move all exceptions into one module
2017-05-22 16:51:11 -03:00
Michael Pöhn
a8420817cb
fix calling vagrant global-status
2017-05-22 17:40:19 +02:00
Michael Pöhn
592a292847
use stable version of priv ext instead of latest when testing in makebuildserver
2017-05-22 16:04:30 +02:00
Hans-Christoph Steiner
84dc415837
jenkins-build-makebuildserver: retry git fetch until it works
...
It seems that gitlab gives 500 errors a bit too frequently, so keep
retrying the `git pull` until it works so that there isn't a error email
sent out over failed pulls.
2017-05-22 16:04:30 +02:00
Hans-Christoph Steiner
780b0e9502
Merge branch 'indexing-fixes' into 'master'
...
indexing fixes
Closes #303
See merge request !270
2017-05-22 09:32:39 +00:00
Hans-Christoph Steiner
0345d2c79f
update: find aapt when it is not in the PATH
...
This is some very messy logic built up since 2010. This will all go away
once we have a python3 version of androguard available.
The removed imports and `dir(APK)` is to silence pyflakes
closes #303
2017-05-18 17:21:05 +02:00
Hans-Christoph Steiner
c348186ad6
update: fix fastlane scraping, it uses a subdir called 'images'
...
https://commons.wikimedia.org/wiki/File:GetChromium_FeatureGraphic_1024x500.png
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
77a65be9db
update: skip packages without metadata in index V1
...
If a package does not have a matching "app" metadata file, then it should
not be added to the index. index V0 already does this properly.
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
a17b95a860
update: improved logging
...
the aapt warnings were very verbose, and the other now includes the
file name type for screenshots, rather than the filename twice.
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
5d705452f5
update: allow repo files to use _ in the file names
...
_ is a valid character for Java package names, so it should also work in
the repo file naming scheme. This makes it so it only splits the file
name based on the last _.
2017-05-18 16:13:17 +02:00
Michael Pöhn
