#archlinux32 | Logs for 2019-07-19

[00:27:32] -!- isacdaavid has quit [Ping timeout: 245 seconds]
[00:38:08] -!- samantaz has quit [Ping timeout: 258 seconds]
[00:40:11] -!- samantaz has joined #archlinux32
[01:06:53] -!- thePiGrepper has quit [Ping timeout: 258 seconds]
[04:11:50] -!- samantaz has quit [Read error: Connection reset by peer]
[05:54:56] -!- bill-auger has quit [Ping timeout: 268 seconds]
[05:55:52] -!- bill-auger has joined #archlinux32
[07:05:15] -!- abaumann has joined #archlinux32
[07:05:15] <buildmaster> Hi abaumann!
[07:05:15] <buildmaster> !rq abaumann
[07:05:16] <phrik> buildmaster: <abaumann> sometimes I get the feeling that the only thing developing fast on Linux is the number of new bugs.
[07:05:20] -!- abaumann has quit [Client Quit]
[07:12:31] -!- abaumann has joined #archlinux32
[07:12:31] <buildmaster> Hi abaumann!
[07:12:31] <buildmaster> !rq abaumann
[07:12:32] <phrik> buildmaster: <abaumann> YASMSBOP: yet another silly make system based on python
[07:35:34] <abaumann> mmh. I see i686/binutils hopping from slave to slave..
[07:35:45] <abaumann> ..producing thousands of test errors
[07:48:30] -!- prurigro has quit [Read error: Connection reset by peer]
[07:48:32] -!- dopsi has quit [Read error: Connection reset by peer]
[07:48:47] -!- dopsi has joined #archlinux32
[07:49:08] -!- prurigro has joined #archlinux32
[08:07:07] -!- davor has quit [Ping timeout: 244 seconds]
[08:08:02] -!- davor has joined #archlinux32
[08:26:33] <abaumann> yes, again I have an endless buildloop on eurobuild6-1 with i686/binutils
[08:27:21] <abaumann> also building failing packages over and over again (like mdadm, virtualbox) doesn't make sense.
[08:27:46] <abaumann> The build order should be recalculated on errors, giving errornous packages a lower priority
[08:38:15] <abaumann> binutils builds and uploads correctly, the build number is up to 74 now: pool/binutils-2.32-2.74-i686.pkg.tar.xz
[08:38:25] <abaumann> so, I stop all slaves..
[08:39:58] -!- abaumann has quit [Quit: leaving]
[08:44:49] -!- deep42thought has joined #archlinux32
[08:44:49] <buildmaster> Hi deep42thought!
[08:44:49] <buildmaster> !rq deep42thought
[08:44:50] <phrik> buildmaster: * deep42thought listens carefully, but doesn't hear any bells ringing over here
[08:45:05] <deep42thought> erroneous packages _do_ get a lower priority
[08:45:14] <deep42thought> (e.g. are scheduled at the end of the queue)
[08:45:28] <deep42thought> however, building binutils over and over is an error of the toolchain-extra-logic
[08:45:30] <deep42thought> :-(
[09:16:59] <deep42thought> binutils should not be handed out
[09:17:04] <deep42thought> gcc should be handed out
[09:17:14] <deep42thought> and once that is returned, binutils should be built
[09:17:25] <deep42thought> and then gcc should be built again ...
[09:29:00] <deep42thought> arrrgh - for some reason, the toolchain order chokes on handing out i686 packages to pentium4 slaves (which should not be an issue at all)
[09:55:17] <deep42thought> I started a discussion about improvements for get-assignment in the forums: https://bbs.archlinux32.org
[09:55:19] <phrik> Title: scheduling of builds in get-assignment / Building / Arch Linux 32 Forums (at bbs.archlinux32.org)
[11:15:29] <deep42thought> aha! mdadm is in base, so it blocks *everything*
[11:31:14] -!- abaumann has joined #archlinux32
[11:31:15] <buildmaster> Hi abaumann!
[11:31:15] <buildmaster> !rq abaumann
[11:31:16] <phrik> buildmaster: <abaumann> that's what happens if fixing a bug is faster than checking out the git repo of the linux kernel.
[11:31:22] <deep42thought> Hi abaumann!
[11:31:26] <abaumann> Hi deep42thought
[11:31:32] <abaumann> Arrived in my outdoor office. :-)
[11:31:41] <abaumann> aha. mdadm.
[11:31:46] <deep42thought> :-)
[11:31:51] <abaumann> let me see if I can do something useful there..
[11:38:02] <deep42thought> it fails on x86_64, too
[11:38:47] <abaumann> ah, cool.
[11:38:57] * deep42thought creates a bugreport
[11:39:01] <abaumann> :-)
[11:39:56] <abaumann> cc1: all warnings being treated as errors
[11:40:09] <abaumann> to make eli very happy :-)
[11:41:01] <deep42thought> !bug 63228
[11:41:02] <phrik> https://bugs.archlinux.org
[11:41:59] <abaumann> so, we do the full regression for upstream ;-)
[11:42:16] <deep42thought> yes, we do
[11:42:31] <abaumann> I think it's a simple matter to remove the Werror (as this is a gcc 9 feature)
[11:42:33] <deep42thought> that's why I wrote a script to generate those bug reports
[11:42:49] <abaumann> handy :-)
[11:43:02] <deep42thought> ok, let's do it (and get the 2k haskell-* packages into the loop again)
[11:44:35] <abaumann> I like the test framework of mdadm: it must _not_ run in a chroot and it _must_ run as root :-)
[11:44:45] <deep42thought> whoah
[11:44:55] <abaumann> ok, quite understandable, actually.
[11:45:07] <deep42thought> but it does not come with 'curl | bash'?
[11:45:26] * abaumann greps for curl in mdadm sources
[11:45:32] <abaumann> no :-)
[11:45:35] <deep42thought> and wget and ncat and nc
[11:45:54] <abaumann> removing -Werror in Makefile works with warning.
[11:45:56] <deep42thought> and `eval "$(printf '%s' c u r l)"`
[11:46:06] <abaumann> I'll fix that temporarily in packages32
[11:46:10] <deep42thought> yes, thanks
[11:46:12] <abaumann> and reference it
[11:48:25] -!- deep42thought has quit [Remote host closed the connection]
[11:48:53] -!- deep42thought has joined #archlinux32
[11:48:53] <buildmaster> Hi deep42thought!
[11:48:53] <buildmaster> !rq deep42thought
[11:48:54] <phrik> buildmaster: <deep42thought> golang-asvnp-acwduac-sefc-evfsx-vb4fs-sefvcy is broken (says me)
[12:03:52] * abaumann wonders on which RAID he should test mdadm 32-bit
[12:50:12] <abaumann> so, after playing with buildmaster-slow.log I cannot see slow queries..
[12:50:42] <abaumann> ..one problem though is that by using shell, mysql is started for every single command, effectively closing the connection to the database
[12:51:27] <abaumann> I leave the slow log on, it's written to /var/lib/mysql/buildmaster-slow.log
[12:51:36] <abaumann> maybe we spot there something running for hours ;-)
[12:53:55] <abaumann> there are some false alarms in there: not every query not having an index is a bad query. on small or partitioned tables a full table scan might be more efficient
[13:04:07] <abaumann> mmh get-package-updates is waiting for something
[13:15:51] <abaumann> | 160010 | root | localhost | buildmaster | Query | 3191 | Sending data | INSERT IGNORE INTO `bl` (`arch`,`pkgbase`) SELECT `a_bp`.`architecture`,`a_ps`.`pkgbase` FROM `packa | 0.000 |
[13:16:07] <abaumann> so, this runs since 53 minutes..
[13:16:24] <deep42thought> sounds like the blacklisting
[13:16:36] <abaumann> yes, it feeds the /tmp/*blacklist file into the database
[13:16:53] <abaumann> 331 /tmp/tmp.get-package-updates.5NuQRZfMjd/black-listed
[13:16:57] <abaumann> only 331 packages
[13:17:07] <deep42thought> those are the explicitely blacklisted ones
[13:17:12] <deep42thought> I think
[13:17:16] <abaumann> cannot take that long.
[13:17:37] <abaumann> so, maybe there is a lock somewhere: a package overlapping between the blacklist and some database queries
[13:18:00] <abaumann> show processlist doesn't show anything in parallel, but the casual get-assignments
[13:18:05] <abaumann> and the ssh-logs
[13:18:38] <deep42thought> the long-running part is hidden in `calculate_blacklisted_packages`
[13:19:31] <abaumann> so, for every package on the blacklist it has to go through all packages basically.
[13:19:47] <deep42thought> the logic is transposed to that
[13:20:07] <abaumann> given that CPU is very high there is some complex query running there :-)
[13:20:12] <deep42thought> yes
[13:20:20] <abaumann> good show processlist is cutting off the select..*grmpf*
[13:20:43] <deep42thought> there is a huge join to decide which install_targets are unavailable
[13:20:57] <deep42thought> and those are used to see which packages are dependent on blacklisted packages
[13:21:04] <deep42thought> which are then added to the blacklist
[13:21:11] <deep42thought> and then everything starts all over
[13:21:13] <deep42thought> again
[13:21:31] <abaumann> I was hoping for an index missing there..
[13:21:59] <abaumann> ah.. show full processlist
[13:24:10] <abaumann> if there is one index missing in that join, then I know what's so slow :-)
[13:24:17] <deep42thought> :-)
[13:32:05] <abaumann> insert into ssh_log takes 20 seconds.
[13:32:10] <deep42thought> :-/
[13:32:18] <abaumann> so for every slave trying to do something you get a 20 second penalty :-)
[13:32:40] <deep42thought> this query is not *that* complex
[13:32:47] <abaumann> but: inserting into a big table should not be slower than inserting into a small table
[13:33:04] <abaumann> this is a classical case for a table being ordererd by date
[13:33:30] <abaumann> so basically, the physical order on disk should be sorted by timestamp, because that's the most frequent way you access the data
[13:33:52] * abaumann digs in the mysql manual for the right mysql-spells
[13:33:56] <deep42thought> it uses `insert ignore` - maybe *that* slows it down?
[13:34:12] <abaumann> it should _not_ be an upsert
[13:34:21] <abaumann> otherwise you have a select/insert penalty
[13:34:50] <abaumann> "Use the INSERT IGNORE command rather than the INSERT command. If a record doesn't duplicate an existing record, then MySQL inserts it as usual. If the record is a duplicate, then the IGNORE keyword tells MySQL to discard it silently without generating an error."
[13:35:03] <abaumann> I really like this kind a generic advice of non-db-people!
[13:35:15] <deep42thought> the ignore is superfluid - there is no unique key being inserted anyways
[13:35:28] <abaumann> if you _know_ you cannot have duplicates, then it's better to insert and to fail if it exists.
[13:35:52] <deep42thought> the table does not error on duplicates
[13:36:06] <deep42thought> there is literally only the id (primary key) - which is auto_increment
[13:36:14] <abaumann> ok.
[13:36:48] -!- davor_ has joined #archlinux32
[13:37:16] -!- davor has quit [Ping timeout: 272 seconds]
[13:37:17] davor_ is now known as davor
[13:37:19] <abaumann> this database is not behaving like a proper database
[13:37:35] <abaumann> I see simplistic queries being held up by something for 20 seconds.
[13:37:44] <abaumann> for instance updating a row in slaves
[13:39:38] <abaumann> I smell hot spots..
[13:43:59] <abaumann> I think this behaviour was only while get-package-updates was running. Now things look normal.
[13:44:27] <deep42thought> get-package-updates should create locks on almost all tables
[13:45:09] <abaumann> and ssh-log will get slow..
[13:45:12] <abaumann> seems normal.
[13:48:40] <abaumann> yeah, usual database debugging stuff.. hunting wrong trails.. :-)
[13:57:35] <abaumann> I increased some wait for return-assignment times for build slaves
[13:57:44] <abaumann> it seems more slaves are doing now something..
[13:57:48] <deep42thought> so they won't line up, you mean?
[13:58:05] <abaumann> I always had 3-4 slaves just waiting on my eurobuild6
[13:58:11] <deep42thought> ok
[13:58:22] <abaumann> 15->60 + rand(30)
[13:58:31] <deep42thought> ok
[14:18:21] <deep42thought> the greater number of working build slaves can also be due to mdadm now being built
[14:18:34] <deep42thought> (I mean "finished building")
[14:18:43] <abaumann> that's true
[14:19:00] <deep42thought> anyways: waiting 1 minute will not hurt too much ;-)
[14:19:11] <abaumann> no, especially given the build time of packages.
[14:36:56] <buildmaster> pentium4/lsof is broken (says buildknecht): https://archlinux32.org
[14:41:09] <buildmaster> pentium4/tcp-wrappers are broken (says eurobuild6-3): https://archlinux32.org
[14:49:12] <buildmaster> pentium4/lockdev is broken (says eurobuild6-5): https://archlinux32.org
[15:01:23] -!- MannerMan has quit [Quit: MannerMan]
[15:28:58] <abaumann> full selects over mirrors and ssh_log?
[15:29:09] <deep42thought> abaumann: do you have any interest in creating an i486 iso build script?
[15:29:26] <deep42thought> where do we do full selects?
[15:29:29] <abaumann> oeh. in theory it should not be that different from the i686 one
[15:29:34] <abaumann> good question.
[15:29:51] <abaumann> either it's a mysldump showing up in the slow queries
[15:30:08] <abaumann> I doubt any script or php code does a select on million of rows
[15:30:23] <deep42thought> you don't know me properly, it seems :-D
[15:30:36] <abaumann> :-)
[15:31:45] <abaumann> I have a much more worriesome behaviour. all return-assignment are blocked by get-package-updates
[15:31:54] <abaumann> that one takes at least one hour on every invocation
[15:32:13] <deep42thought> I think, this is written in the lock files
[15:32:19] <deep42thought> both lock "the build list"
[15:32:39] <deep42thought> I don't really remember the logic behind that, though
[15:32:56] <deep42thought> I think, we had concurring access and it left the database in a funny state ...
[15:33:01] <abaumann> yep
[15:33:21] <abaumann> I just fetch slow queries and put them into an analyze expression
[15:33:28] <abaumann> this usually is quite informative
[15:33:35] <deep42thought> yeah, sounds good
[15:34:00] <abaumann> I'm all in for cheap optimizations :-)
[15:43:05] <deep42thought> ok, the torrent for the iso works, the magnet link does not
[15:43:44] -!- abaumann has quit [Remote host closed the connection]
[15:54:32] -!- deep42thought has quit [Quit: Leaving.]
[16:46:41] <buildmaster> pentium4/gens-gs are broken (says eurobuild6-4): https://archlinux32.org
[17:17:10] <buildmaster> pentium4/qtpass are broken (says eurobuild6-4): https://archlinux32.org
[17:35:01] <buildmaster> pentium4/pdflib-lite is broken (says eurobuild6-4): https://archlinux32.org
[17:44:50] -!- thePiGrepper has joined #archlinux32
[18:07:57] -!- isacdaavid has joined #archlinux32
[18:10:57] -!- isacdaavid has quit [Read error: Connection reset by peer]
[18:50:35] <buildmaster> pentium4/tap-plugins are broken (says eurobuild6-3): https://archlinux32.org
[19:25:12] <buildmaster> pentium4/wkhtmltopdf is broken (says eurobuild3): https://archlinux32.org
[19:44:20] <buildmaster> pentium4/jxrlib is broken (says eurobuild6-4): https://archlinux32.org
[19:58:21] -!- deep42thought has joined #archlinux32
[19:58:22] <buildmaster> Hi deep42thought!
[19:58:22] <buildmaster> !rq deep42thought
[19:58:23] <phrik> buildmaster: <deep42thought> I have the impression, their main operating area is television via cable
[20:13:54] <buildmaster> pentium4/libidn is broken (says eurobuild6-2): https://archlinux32.org
[21:00:07] <buildmaster> pentium4/valgrind is broken (says nlopc46): https://archlinux32.org
[21:14:42] -!- samantaz has joined #archlinux32
[21:25:41] <slacka123> why this there a nss 3.44-1.1 in testing? was there a problem with .0 ?
[21:33:12] <deep42thought> slacka123: probably we used a too big cannon to shoot down some sparrows
[22:05:12] <deep42thought> abaumann: you did the right thing with mdadm, see: https://bugs.archlinux.org
[22:05:13] <phrik> Title: FS#63228 : [mdadm] fails to build in clean chroot due to some alignment issues (at bugs.archlinux.org)
[22:12:36] -!- deep42thought has quit [Quit: Leaving.]
[22:24:08] <buildmaster> pentium4/qt-gstreamer is broken (says eurobuild6-2): https://archlinux32.org
[22:51:14] -!- thePiGrepper has quit [Ping timeout: 248 seconds]
[22:54:09] <buildmaster> pentium4/shaderc is broken (says eurobuild6-1): https://archlinux32.org
[23:35:26] <buildmaster> pentium4/libvterm is broken (says nlopc43): https://archlinux32.org
[23:43:12] -!- samantaz_ has joined #archlinux32
[23:44:02] -!- samantaz has quit [Ping timeout: 248 seconds]
[23:57:02] <buildmaster> pentium4/adns are broken (says nlopc43): https://archlinux32.org