A great deal of background on cross building in Debian/Ubuntu/Linaro is given on CrossBuilding. This page assumes you are familiar with the information on that page.

  • Launchpad Entry: UbuntuSpec:

  • Created:

  • Contributors:

  • Packages affected:

Summary

This spec summarises the processes, changes and tools needed for a good cross-building experience.

CrossBuilding contains a great deal of background on cross building in Debian/Ubuntu/Linaro. This Spec assumes you are at least somewhat familiar with the information on that page.

Related Specifications:

Release Note

These tools allow developers to cross-build packages against the archive, and to produce small distributions in variant flavours, given suitable cross-tools. They also support a cross-autobuilder for long term cross-building QA.

Rationale

There are several quite different cross-building scenarios, and many different approaches and corresponding tools. Fundamentally these are either a) classical cross-aware cross-building, or b) using emulation, either for everything, or for some parts. There are many variation on these themes to do with usuing sysroot or not, using multiarch or not, whether builds are self-hosting or not. Debian/Ubuntu currently supports classical cross-aware building, without using sysroot, and builds are not self-hosting.

All these options have pros and cons to do with speed, simplicity, maintainability, developer effort and use cases. See 'Use cases' for some examples. We do not have the resource to do everything, so will continue to focus effort for long-term gain and and on the set of use-cases are users have highlighted as important. That means fixing existing systematic problems in packaging and build tools, and using/re-using existing tools as much as possible. Flexibility in the tools is necesary to cover the various use-cases people have.

Bootstrapping a new port from scratch is a much more involved problem than building packages in a distribution context. It is not currently supported in Debian/Ubuntu/Linaro, due to cyclic dependencies, but it is useful for building a new flavour or a new ABI/arch port, and for test purposes. Cross-Cyclic build-dependecnies have been anlysed and work is underway to untangle these packages, fix tools to underastand the changes, and generate tools to use them.

Despite cross-aware cross-building being seen as old fashioned in some quarters, and it requiring more work in packages to get wide coverage, we still believe it is a good model within a binary distribution, because it supports all architectures (doesn't rely on Qemu), is repeatable, and requires things to be done properly, and fixed once, in the source. It improves the quality and correctness of build systems generally, in the same way that building for 12 architectures and multiple kernels does. Using qemu emulation or remote shell processing avoids solving a number of cross-building problems directly, but it only works for supported target architectures, very new architectures are usually not supported. Cross-aware cross-building supports any target, inlcuding new ones that don't exist yet. This is still important. Emulation is also very slow unless you restrict it carefully to things which are not the bulk of operations.

Users want to interface with cross-building with relatively high level tools where they say things like 'build me this package, using this toolchain, against this repository', the latter two items generally being configured once. However the actual process of doing a cross-build involves several discrete steps, detailed later, and tools which aid cross-building need to be able to do those steps one by one, with adequate control of which repositories are used, the environment builds are run in, and where results end up. We have tools for most of these steps but a couple still need work to provide general solutions.

User stories

Fred is working on a package. It is not a trivial piece of software and thus takes a while to build. It is very convenient for him to cross-build rather than native build because it is a lot faster (factor of 6-10 is typical), and if you do 20 builds a day it really matters whether each build took 5 mins or an hour. In this case classical cross-building is most appropriate because it is fastest. Any extra work needed to make the package cross-build properly only needs to be done once.

Now Fred is working on several interdependent packages, which are not in the main distribution. He needs his tools to use the locally-(cross)built packages when satisfying dependencies as well as the main repository. So it must be easy to specify which repository to use for uploads and which for dependency satisfaction.

Nimrod wants to optimise some (or all) packages for a particular target to get maximum performance from his hardware. He needs to rebuild those packages using a flavoured cross-compiler. He can do this against distribution libraries.

Now Nimrod wants to run the same packages on older hardware which doesn't support some features assumed by the default distro build. He needs to rebuild everything with a flavoured compiler targeting the hardware, ensuring that there are no unwanted instructions remaining in the completed packages.

XinHui wants to build a small distro for a device unsuited to native building. She needs to be able to break dependency loops in order to recursively build the whole system against itself.

Albert wants to know if a particular distribution package is known to cross-build properly or not, and all the users above also explicitly or implicitly want to know this. A cross-autobuilder which shows the current state of cross-buildability of the archive would make this information known, and make it easy to find packages that one cares about that need fixing.

Assumptions

We are always building Debian packages, either from local sources or repository-provided sources. We are not catering for source-package modifications at build-time (beyond those already supported using mechanisms like dpkg-vendor and deb-build-opts) - any such modifications must be done to the source before the build, and repository prioroties used to control mixing.

Design

Tools are needed to do the following sub-tasks, which can be combined to do the necessary overall tasks. These low-level tools will be combined by higher-level tools to hide complexities for typical use cases.

  1. Easily create (and maintain) a cross-building environment
  2. Execute a cross-build. (packages must be capable of being cross-built for this to work)
  3. Determine cross-dependencies necessary for a build, from repositories, either remote or local
  4. Install cross-dependencies
  5. Generate cross-installable packages where multiarch-ready packages are not available. (closely tied to 4)
  6. Calculate build-ordering for building multiple packages and bootstrapping
  7. Upload results to a repository: either or both of a 'results' repository and a 'dependency satifaction' repository.
    • This may actually the hardest part to get right. Every build results in packages which might also be dependencies of other builds. They have to end up somewhere that the next build can find them. That should always be a repository of some sort, even if it's only a local apt-ftparchive tree. Normally it should be the target repository for which the builds are being run, but this may have delayed uploads, signing requirements or even rebuild delays, so may not always be useful quickly enough. In this case a local repository (as maintained internally by xdeb) is also needed.

Metadata and package structure is needed to make these tools reliable:

  1. It must be possible to determine cross-dependencies in order to install them. Multiarch annotations provide good defaults for this information, but exception to that need to be encoded in build-dependencies. Where it is not present either tool heuristics or brute force most be used instead.
  2. Packages must support cross-building (run (and possibly generate) build not target binaries, skip tests)
  3. Packages which depend on themselves (in an arch-dependent way) to build must support a staged build to be bootstrappable.

Implementation

There are various things needed:

  • Improved metadata in packages
    • Multiarch status on packages and exception labels ('native'/'same'/'both') on build-dependencies so that cross-dependencies can be accurately calculated
    • Flavouring metadata so that library linking compatibility can be checked (i.e. something to to say that this 'armel' package is v7-vfp-thumb2 or v5-arm)
    • Correcting missing dependencies which are only noticed in cross-builds
    • Staging in builds to allow bootstrapping
  • Fixed tools and packages to solve cross-building problems found so far:
    • arch-specific pkg-config (done)
    • libtool changes or packaging improvements to avoid linking native and foreign binaries (not needed with multiarch paths)
    • Correcting use of foreign tools during the build when it should be a native tool
    • Making gobject introspection cross-friendly
  • Automated continuous integration
    • This is important because it's hard to know at any given time what is and isn't working. Anything of toolchains, 'make' systems, build tools, metadata, or state of the archive can prevent cross-builds working. Manually running lots of builds under lots of circumstances is time-consuming and error-prone. Having buildd type infrastruture reporting status online is important to see where the biggest problems are and the current state of the archive in terms of 'crossability'. I envision an autobuilder running builds for Ubuntu and Debian armel and armhf in maybe 3 flavours each to get good coverage. We can start smaller than that, obviously. Website and build-resource is needed for this. Those two things probably on separate machines.
  • Having low-enough level tools to allow crossbuilding in all the various important use-cases. We already have tools for several of these tasks
    • Easily create (and maintain) a cross-building environment:
      • Cross-building can legitimately be done on the main machine or in a chroot. It is generally best done in a chroot for reasons of repeatability, and (pre-multiarch) to avoid polluting the build system with hundreds of 'locally built or obsolete' cross-packages. We should enable both methods of working, whilst recommending building in chroots. Thus tools must not make undue assumptions about the environment -if properly designed it shouldn't matter at all (to the tool) whether it tool is running in a chroot or not. Pdebuild-cross (using multistrap) provides exactly this function of making it easy to say 'make me a cross-building chroot using these cross-tools'. This should be integrated into pbuilder itself in due course, and adjusted to make it more flexible about which tools are used inside the chroot. Using sbuild+schroot snapshots has also been proposed as providing similar but possibly more versatile functionality. This will be investigated. If we get the componentization right then both should work easily and play to their strengths. The environment defines the default repository set being used to satisfy dependencies (in the apt config).
    • Execute a cross-build. (packages must be capable of being cross-built for this to work)
      • dpkg-buildpackage -a<arch> has provided this functionality for some time. Some packages need environment variables setting, but that should be considered a bug to fix. Packages which do not use autotools or cmake or some other cross-aware 'make' system may need significant changes to cross properly.

    • Satisfy Cross-dependencies necessary for a build, from repositories, either remote or local
      • We have several such tools, none entirely satisfactory. (xapt, xdeb, apt-cross) So a significant outcome of this cycle will be a tool or tools to do this. An separate 'xdeb-satisfy-crossdeps' that uses multiarch metadata, falling back to either xdeb's heuristics/black/whitelists or xapt's exhaustive brute-force when metadata is not present. The compementary xdeb-satisfy-deps to ensure the native dependencies are present would also be a good idea. apt-get build-dep will also satisfy this function but will also install all the cross-deps natively, which is just a waste of time.
    • Install cross-dependencies
      • Apt, now that it is architecture-aware, has all the functionality we need so long as dependencies are always in a repository of some sort. With multiarch we just install the foreign version of the package. Without it we use the -cross namespace.
    • Generate cross-installable packages where multiarch-ready packages are not available.
      • Multiarch provides this functionality 'for free'. dpkg-cross provides it when multiarch is not enabled. Unfortunately the two make provision on different paths, (using tuples/triplets respectively). Mixing these paths almost certainly causes problems. We cannot support builds where different versions of packages are installed at once, one using each mechanism. dpkg does not automatically prevent this state because the -cross packages are renamed (see issues). We may not be able to support reliable builds where both old and new paths are used together because it can very easily result in native and foreign binaries being attempted to be linked together. Managing this transition will be awkward. This probably doesn't need to be a separate command (?)
    • Calculate build-ordering for building multiple packages and bootstrapping
      • Xdeb has this functionality internally but it is not currently accessible on its own. Breaking this out from the tool will allow a proper bootstrapper to be built.
    • Upload results to a repository: either or both of a 'results' repository and a 'dependency satifaction' repository.
      • There is no shortage of upload and repository management tools. Making them work so that results always end up in the desired final place and so that previously-build dependencies are available, even in the presence of disposable chroots needs careful thought.

UI Changes

Keeping the existing xdeb interface for the way it currently works, as a user-level all-in-one cross-builder is desireable. Thus other commands will be created for the sub-tasks. I suggest there are all be of the form 'xdeb-subcommand', which gives users good discoverability xdeb-<TAB> being helpful. (and xdeb is a handy name - might as well keep using it even if we radically change the usage). It will probably work out that some of these commands are just a very thin veneer over some other command. (e.g. 'apt-get -oAPT::Architecture=<arch> package').

Suggested command list (subject to revision):

xdeb-satisfydepends [--list] [--cross|--build] <sourcepackage>
xdeb-build <sourcepackage>
xdeb-satisfydepends <package>
xdeb-putresult <sourcepackage>
xdeb-sequence <package list>  
xdeb-chroot create|update [<target arch>] [<config>]  (and login?)

Everything takes an optional target arch, but normally defined in chroot config.

pkg-config has gained architecture-specific variants like arm-linux-gnueabi-pkg-config with DTRT

Code Changes

xdeb will be re-architected to split out the various functions that it would be useful to use separately. Other tools will be used where they already do the right thing or can easily be made to. Keeping the existing useful python module functionality for traversing source trees and generating parse trees seems very sensible, and each command uses those modules.

Move some config into apt itself. (default repos, arches in use) (allow xdeb overrides?)

pkg-config changes

xapt and xdeb get knowledge of multiarch dep-qualifiers

Having examing how xdeb is used in real projects, and the advantages and problems of its current design, I propose this as a more logical and predictable way for it to work:

Repository config

Primary repo config done via apt:

  • deb lines for binaries - default to build machine/chroot entries.
  • deb-src lines for sources - default to build machine/chroot entries
  • preferences for policy - default to build machine/chroot entries

Ignore build machine/chroot entries if --local-config used: use xdeb.cfg entries instead.

Preferentially use implicit local repo unless --disable-local-repo is used, reflects current behaviour and means that local sources will be used by default, and a set of packages with local dependencies works easily without config.

It would be more rigorous to have local repo configured just like remote repo, as above - then use policy for pinning etc. But then less stuff 'just works'. Perhaps this should be set up by install script, to give both automatic sensible defaults, but evenly-exposed config.

For each repo can configure an auto-upload command. How to deal with remote repos with periodic 'incoming' processing? Put all builds in local repo too, as current design?

Change current swamp of implicit priorities, --prefer-apt, --apt-source to:

  • --use-remote-repo|--disable-remote-repo
  • --use-local-repo|--disable-local-repo

Nice and clear what happens, and what will be used. Currently it can be very hard to know what's going on, and why.

Command set

xdeb-build <sourcepkg>:

  • Builds single package. binary deps must be available somewhere (does not recurse).
  • Default mode is --use-remote-repo and --use-local-repo
  • call xdeb-satisfydepends && xdeb-runbuild && xdeb-upload (if upload <repo> is set)

xdeb-satisfydepends <sourcepackage>:

  • check immediate build-deps of foo.
  • Determine which are cross, which native. (multiarch if specified, xdeb internals if not).
  • check availability of native deps in repo(s). Fail if not available.
  • check availability of cross deps in repo(s). Fail if not available.
  • install needed build-arch deps using apt
  • install needed host_arch deps using apt if possible.
  • Fall back to apt-get download+dpkg-cross and then wget+dpkg-cross if apt-install fails. This will be necessary until mutiarch transition is completed

xdeb-runbuild <sourcepackage>:

  • almost just call dpkg-buildpackage -a.
  • do existing setup ENV VAR extras for now, (but we shouldn't be relying on those in long term).

xdeb-sequence <list of sourcepackages>:

  • Essentially equivalent to current xdeb
  • depth-first search as currently, then xdeb-build in order
  • xdeb-sequence --bootstrap to allow extra search of bootstrap space
  • xdeb-sequence --full to do full rebuild ignoring current builds.
  • xdeb-sequence --list to list but not run anything

Unresolved issues

  • Clarify distinction (if any) between local repo and on-disk sources.
  • xdeb-sequence needs more thought on how to ensure -sequence and -satisfydepends agree on what is to be done. Recalculating may be inefficient. Perhaps pass results cache?
  • Exactly how bootstrap/staged-builds dep space will be searched - does it fit in here?

Migration

  • Wiki documentation at CrossBuilding will be updated as things change.

Test/Demo Plan

Prove that we can

1) Build individual packages against a repository

2) Build a set of packages against each other and a repository

3) Build a set of flavoured packages against themselves

4) bootstrap a cyclic set of packages from scratch

Unresolved issues

This spec makes no attempt to deal with modification of source packages in order to build variant packages. That is for a later spec. We will try to avoid doing anything that makes this awkward problem any harder to deal with.

It depends on other Specs for multiarch support and staged builds (for bootstrapping)

There is a tension between supporting self-hosting builds and distro-supported builds when it comes to tools built and used during the build. For bootstrapping and self-hosted builds it is good to build the tool during the build then use it on some local path. This can be accommodated by ensuring it is build for the build arch, not the target arch, but this can be invasive (e.g. when normally the build builds the tool only once, but we need to end up with both a native and a target version, so now it needs building twice). It is easier in a Debian cross-building view of the world if the native tool is supplied by a native dependency (this only works where the tool is propogated into the package and is not a build-time only thing). This scheme cannot accomodate significant version skew between the supplied tool and the version that would have been internally built (in case the behaviour changes). The correct Debian view is that such tools which make it into the final package(s) should really be packaged separately so that they can be a build dependency, then everything is lovely. Where the tool does not make it into the final package(s) then building it at build-time (for the build arch) is acceptable.

Migrating from existing triplet-based cross-paths to new multiarch-based paths is going to cause problems. I am not sure how bad, or exactly how to deal with them. Each package knows whether it is using multiarch paths or not, but a packaging depending on several libraries could easily find some on one path and some on the other. However simply providing both paths could caue problems unless something ensures that old and new versions are not installed at the same time, and I think there is potential to accidentally get native libs linked in (or headers used) accidentally too. How well this transition will owrk from a cross-building POV remains unclear. we'll have to suck it and see. This is one reason why autoated cross-building infrastrucutre is important.

dpkg does not automatically prevent libfoo[armel] and libfoo-armel-cross (potentially of different versions) both being installed at the same time. Pristine chroot building should largely protect against this problem, but if -arch-cross packages are left installed or locally generated during a build then this could arise. Perhaps a check-tool to monitor for this state should be created? Should search paths specify multiarch first and old-triplet second or the other way round? Should -cross packages be made to conflict with the corresponding non-cross package? There was discussion of this on the debian-embedded list earlier this year: http://www.mail-archive.com/[email protected]/msg04940.html

BoF agenda and discussion

  • Get usage scenarios from cross-build tool users - we have ARM ALIP and TI stuff (https://launchpad.net/~tiomap-dev/+archive/release ) What else?

  • Continuous integration testing of cross-builds.
    • Lots of packages don't build
    • Manual testing (and especially re-testing) is dull
    • Autobuilder testing archive cross-coverage. Needs cross-dep checker for sequencing - do we have a separable one?
  • Issues observed from cross-building ALIP packages with xdeb and pdebuild-cross. (see https://spreadsheets.google.com/ccc?key=0AnPR4S1Uev7KdDhHM2RxVGFFY2VQM01MVEJXbTZ3TkE&hl=en&pli=1#gid=0 and https://wiki.linaro.org/CurrentPackageCrossBuildStatus )

    • missing dependencies
    • strip issues with native libs
    • libtool linking in native libs
    • runtime tools: packaged (use native build-dep (circular), or build twice?), unpackaged (build native)
  • Consider multiarch integration and how that affects approach
    • how do we transition from existing paths and mechanisms to multiarch paths/mechanisms?
    • Timescale for change. How long do we carry on supporting existing -cross scheme?
  • Make cross-gobject-introspector
  • Consider pros/cons of one capable, monolithic tool vs a set of integrated 'do one job well' tools:
    • Cross-dep satisfier (xapt, xdeb, apt-cross)
    • build-recurser/seqencer (strip out of xdeb?)
    • package-crosser (dpkg-cross)
    • cross-builder (dpkg-buildpackage -a<arch>)

    • bootstrapper(stage-building tool) (mostly package metadata - see https://wiki.ubuntu.com/Specs/M/ARMAutomatedBootstrap), then teach build-recurser to use

    • source downloader (apt, or already present locally)
    • package uploader (dupload, dput, apt-ftparchive, reprepro, PPAs)
  • management/use of internal/external repositories (can xdeb use an external one?)
  • xdeb issues: caching/rebuild behaviour with --only-explicit, packages that fail to dpkg-cross during dependency satisfaction.
  • pdebuild-cross issues: multistrap vs debootstrap, deps for 'make clean' needed in external chroot.


CategorySpec CrossBuilding

Platform/DevPlatform/Specs/CrossBuilding (last modified 2011-05-23 03:05:03)