Some packages refuse to (re)build without rake

[eli-schwartz]

At least, gdk3 will not do so when bootstrapping quarry.

Building native extensions. This could take a while...
ERROR:  Error installing gdk3-3.2.1.gem:
	ERROR: Failed to build gem native extension.

    current directory: /build/ruby-gdk3/pkg/ruby-gdk3/usr/lib/ruby/gems/2.5.0/gems/gdk3-3.2.1/dependency-check
rake RUBYARCHDIR=/build/ruby-gdk3/pkg/ruby-gdk3/usr/lib/ruby/gems/2.5.0/extensions/x86_64-linux/2.5.0/gdk3-3.2.1 RUBYLIBDIR=/build/ruby-gdk3/pkg/ruby-gdk3/usr/lib/ruby/gems/2.5.0/extensions/x86_64-linux/2.5.0/gdk3-3.2.1
sh: rake: command not found

rake failed, exit code 127

Gem files will remain installed in /build/ruby-gdk3/pkg/ruby-gdk3/usr/lib/ruby/gems/2.5.0/gems/gdk3-3.2.1 for inspection.
Results logged to /build/ruby-gdk3/pkg/ruby-gdk3/usr/lib/ruby/gems/2.5.0/extensions/x86_64-linux/2.5.0/gdk3-3.2.1/gem_make.out

This might not be the only package that no longer cleanly builds after https://lists.archlinux.org/pipermail/arch-dev-public/2018-January/029120.html

...

EDIT: It helps if you don't delete the existing config.pkg/*.yaml :D
Anyway, vte3 has the same issue, let's see what else crops up.

[anatol]

Thanks for info. Yes these two packages need rake make dependency. I added it and was able to build it successfully.

Do you see the issue with other packages?

[eli-schwartz]

Not yet. I got all the way to ruby-debug-ide, which failed because it declares a patched-in dependency on debase but no other package currently needs it, and it is not in the whitelist, and quarry does not add dependencies from config.pkg/*.yaml to the build queue.

So, I've just appended it to the whitelist as a dirty hack, and the mass build is now progressing again.

[anatol]

Yeah, I see ruby-debug-ide uses debase gem https://github.com/ruby-debug/ruby-debug-ide/blob/master/Gemfile but does not declare it in its *.gemspec. It is weird. Maybe upstream should be contacted to clarify this situation.

In a short-term just add a dependency to its config.pkg with a comment why it been added + this issue number.

PS here is upstream issue ruby-debug/ruby-debug-ide#92

[eli-schwartz]

Uhhh, there's already a patch in quarry :D and the package has a depends+=(ruby-debase) and a prepare() that patches it into the gemfile.

But, the build queue does not realize that it should build debase, and therefore makechrootpkg fails with an error: target not found: ruby-debase

This is why I am saying, if a package already declares a config.pkg dependency on a package that is a valid ruby package, it should be added to the build queue alongside packages in the whitelist.
Otherwise the package is required, but not built with the native dependencies.

[anatol]

there's already a patch in quarry :D and the package

Oops missed that.

if a package already declares a config.pkg dependency on a package that is a valid ruby package, it should be added to the build queue alongside packages in the whitelist.

That is how quarry worked from the beginning. Are you saying that is not true anymore? If so then it is a bug and should be fixed.

[eli-schwartz]

It... doesn't seem to be true? As I said, ruby-debase is not being built.

quarry/lib/quarry.rb

Lines 564 to 579 in 2e4b609

	while pkg = packages_to_generate.last
	version = slot_to_version(*pkg)
	spec = package_spec(pkg[0], version)
	upfront_deps = [] # packages should be processed before 'pkg'
	for d in spec.runtime_dependencies
	s = dependency_to_slot(d)
	key = [d.name, s]
	if packages_to_generate.include?(key)
	# if dependency has to be generated, do it before 'pkg'
	packages_to_generate.delete(key)
	upfront_deps << key
	elsif not existing_packages[key]
	upfront_deps << key
	end
	end

And I stuck a bunch of debugging prints to see what is going on at various stages, this is what I get -- notice that debase is not mentioned anywhere. AFAICT dependencies are only added from spec.runtime_dependencies and not the config.pkg:

Queueing ["unicorn", nil] to build...
building OFFICIAL_PACKAGES cache...
Not building ["unicorn", nil] as package is in the official repos
sync root chroot in case next package needs the package we just build
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 quarry is up to date
:: Starting full system upgrade...
 there is nothing to do
Queueing ["treetop", nil] to build...
Not queueing ["polyglot", nil] because it is already in the repos
Not building ["treetop", nil] as package is in the official repos
sync root chroot in case next package needs the package we just build
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 quarry is up to date
:: Starting full system upgrade...
 there is nothing to do
Queueing ["tilt", nil] to build...
Not building ["tilt", nil] as package is in the official repos
sync root chroot in case next package needs the package we just build
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 quarry is up to date
:: Starting full system upgrade...
 there is nothing to do
Queueing ["thor", nil] to build...
Not building ["thor", nil] as package is in the official repos
sync root chroot in case next package needs the package we just build
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 quarry is up to date
:: Starting full system upgrade...
 there is nothing to do
Queueing ["thin", nil] to build...
Not queueing ["rack", nil] because it is already in the repos
Not queueing ["eventmachine", nil] because it is already in the repos
Not queueing ["daemons", nil] because it is already in the repos
Not building ["thin", nil] as package is in the official repos
sync root chroot in case next package needs the package we just build
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 quarry is up to date
:: Starting full system upgrade...
 there is nothing to do
Queueing ["ruby-debug-ide", nil] to build...
Not queueing ["rake", nil] because it is already in the repos
Building ["ruby-debug-ide", nil]...
==> Synchronizing chroot copy [/var/lib/quarry/quarry/work/chroot/root] -> [quarry]...done
==> Making package: ruby-ruby-debug-ide 0.6.1-1 (Fri Feb 16 14:20:09 EST 2018)
==> Retrieving sources...
  -> Found ruby-debug-ide-0.6.1.gem
  -> Found patch
==> Validating source files with sha1sums...
    ruby-debug-ide-0.6.1.gem ... Passed
    patch ... Passed
==> Making package: ruby-ruby-debug-ide 0.6.1-1 (Fri Feb 16 14:20:11 EST 2018)
==> Checking runtime dependencies...
==> Installing missing dependencies...
error: target not found: ruby-debase
==> ERROR: 'pacman' failed to install missing dependencies.
==> ERROR: Build failed, check /var/lib/quarry/quarry/work/chroot/quarry/build
Traceback (most recent call last):
	4: from ./lib/update.rb:20:in `<main>'
	3: from /var/lib/quarry/quarry/lib/quarry.rb:636:in `build_packages'
	2: from /var/lib/quarry/quarry/lib/quarry.rb:574:in `build_package'
	1: from /var/lib/quarry/quarry/lib/quarry.rb:574:in `chdir'
/var/lib/quarry/quarry/lib/quarry.rb:581:in `block in build_package': The binary package was not built: ruby-ruby-debug-ide-0.6.1-1-x86_64.pkg.tar.xz (RuntimeError)

• Note that "not building ... as package is in the official repos" is the warning when we skip running build_package
• "Not queueing ... because it is already in the repos" is what happens when a dependency key is rejected from getting added via upfront_deps << key.
• "Queueing ... at the head of the stack" indicates a dependency key from spec.runtime_dependencies was already in packages_to_generate and had to be deleted and re-added
• "Adding ... to the queue" is the same but for keys that are not yet in packages_to_generate.

[anatol]

ruby-debase is not being built.

Then it is definitely a Quarry bug.

The dependency list is built by package_with_changed_dependencies() that calls generate_dependency_list() that has this snippet

if config and config['depends']
  dependencies = config['depends'] + dependencies
end

The dependencies are lost somewhere along the way.

[eli-schwartz]

Hmm, looks like package_with_changed_dependencies is modifying the existing_packages in place, and adding that to the list of packages that need to be built (with dependencies included).

This only works if the package has been built before without the added dependency from config.pkg, which is why I'm seeing this error -- I am attempting to build all packages from scratch.

I discovered this when it turned out pp changed_dep_packages returned nothing at all, despite this being obviously wrong.