#archlinux32 | Logs for 2020-01-16

[00:51:01] -!- isacdaavid has quit [Read error: Connection reset by peer]
[02:03:55] <buildmaster> i686/xsecurelock is broken (says nlopc46): https://archlinux32.org
[02:05:01] <buildmaster> pentium4/xsecurelock is broken (says eurobuild6-2): https://archlinux32.org
[02:28:14] -!- thePiGrepper has joined #archlinux32
[02:31:22] -!- bill-auger_ has quit [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.]
[02:31:48] -!- bill-auger has joined #archlinux32
[02:53:22] -!- isacdaavid has joined #archlinux32
[02:59:55] -!- isacdaavid has quit [Read error: Connection reset by peer]
[03:06:21] -!- isacdaavid has joined #archlinux32
[03:51:31] <buildmaster> i486/python-autobahn is broken (says eurobuild6-7-i486): https://archlinux32.org
[03:53:37] -!- thePiGrepper has quit [Ping timeout: 260 seconds]
[03:55:09] -!- thePiGrepper has joined #archlinux32
[04:27:27] -!- thePiGrepper has quit [Ping timeout: 260 seconds]
[04:59:06] -!- isacdaavid has quit [Remote host closed the connection]
[06:03:58] -!- tyzoid has quit [Ping timeout: 260 seconds]
[06:06:27] -!- tyzoid has joined #archlinux32
[06:06:27] <buildmaster> Hi tyzoid!
[06:06:27] <buildmaster> !rq tyzoid
[06:06:28] <phrik> buildmaster: <tyzoid> bugs are just unintended features
[06:45:41] -!- Dimtree has quit []
[07:06:12] <buildmaster> i486/emscripten is broken (says nlopc46-i486bs0): https://archlinux32.org
[07:11:38] <buildmaster> i486/glfw is broken (says nlopc46-i486bs0): https://archlinux32.org
[07:24:15] <buildmaster> pentium4/emscripten is broken (says eurobuild6-1): https://archlinux32.org
[07:32:27] -!- abaumann has joined #archlinux32
[07:32:28] <buildmaster> Hi abaumann!
[07:32:28] <buildmaster> !rq abaumann
[07:32:29] <phrik> buildmaster: <abaumann> in 2019 it seems hard to get simple things working.. like filesystem actually being mounted after a reboot :->
[07:33:14] <abaumann> deep42thought: I finally managed to burn a CD. I had an issue installing 'joe' (unknown trust for my sign key). But installing software on the RAMdisk of the ISO is not really a supported scenario, I think.
[07:33:29] <abaumann> Boots fine on a real i686 without SSE2 :-)
[07:34:42] -!- abaumann has quit [Client Quit]
[08:23:10] <girls> abaumann: thanks for testing! Some explanation for my caution: I saw in the iso-build log, that mkinitcpio ran during build and since we dropped all the "setarch"s, it ran under x86_64. I was afraid, this might create a ramdisk unsuitable for i686.
[10:01:51] <buildmaster> i686/m4ri is broken (says nlopc46): https://archlinux32.org
[10:04:27] -!- DepositePirate has quit [Quit: Connection reset by beer]
[10:04:43] -!- DepositePirate has joined #archlinux32
[10:10:41] -!- DepositePirate has quit [Quit: Connection reset by beer]
[10:11:03] -!- DepositePirate has joined #archlinux32
[10:21:24] -!- DepositePirate has quit [Quit: Connection reset by beer]
[10:21:36] -!- DepositePirate has joined #archlinux32
[10:22:46] -!- DepositePirate has quit [Client Quit]
[10:22:57] -!- DepositePirate has joined #archlinux32
[10:36:50] -!- davor_ has joined #archlinux32
[10:39:14] -!- davor has quit [Ping timeout: 240 seconds]
[10:39:14] davor_ is now known as davor
[10:52:53] -!- abaumann has joined #archlinux32
[10:52:53] <buildmaster> Hi abaumann!
[10:52:53] <buildmaster> !rq abaumann
[10:52:54] <phrik> buildmaster: <abaumann> well, we found out that those disks _ARE_ actually slow disks then :-)
[10:52:59] <girls> Hi abaumann!
[10:53:08] <abaumann> hi girls!
[10:54:09] -!- deep42thought has joined #archlinux32
[10:54:09] <buildmaster> Hi deep42thought!
[10:54:09] <buildmaster> !rq deep42thought
[10:54:10] <phrik> buildmaster: <deep42thought> because you're a pessimist with no hope - or you're a realist afraid of change ... pick your evil
[10:54:29] <abaumann> moto of the day :-)
[10:54:34] <deep42thought> :-D
[11:28:03] -!- DepositePirate has quit [Ping timeout: 240 seconds]
[11:30:21] -!- DepositePirate has joined #archlinux32
[11:59:35] -!- abaumann has quit [Quit: leaving]
[12:03:44] -!- bill-auger has quit [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.]
[13:35:37] -!- thePiGrepper has joined #archlinux32
[13:43:17] -!- bill-auger has joined #archlinux32
[14:09:31] -!- davor has quit [Ping timeout: 272 seconds]
[14:11:59] -!- davor has joined #archlinux32
[14:23:51] -!- Cthulu201 has quit [Quit: Nowhere special. I always wanted to go there.]
[15:08:04] -!- deep42thought has quit [Quit: Leaving.]
[15:20:39] <buildmaster> i686/vibe-d is broken (says eurobuild6-2): https://archlinux32.org
[15:25:53] <buildmaster> pentium4/vibe-d is broken (says eurobuild6-2): https://archlinux32.org
[15:41:01] <buildmaster> i686/faust is broken (says eurobuild6-2): https://archlinux32.org
[15:41:03] <buildmaster> pentium4/faust is broken (says nlopc46): https://archlinux32.org
[15:53:35] <buildmaster> i486/ucblogo is broken (says nlopc46-i486bs0): https://archlinux32.org
[16:19:39] -!- Cthulu201 has joined #archlinux32
[16:56:11] -!- crns_ has joined #archlinux32
[16:56:27] -!- crns has quit [Ping timeout: 260 seconds]
[17:06:19] crns_ is now known as crns
[17:06:56] -!- crns has quit [Quit: bye]
[17:07:16] -!- crns has joined #archlinux32
[18:36:40] <nit-picker> dependencies of xorg-server-common-1.20.5-2.4-i486.pkg.tar.xz (built on 2019-09-14) differ between the package and our database
[18:37:20] <buildmaster> i486/xf86-input-libinput is broken (says nlopc46-i486bs0): https://archlinux32.org
[18:42:43] <buildmaster> i486/vis are broken (says nlopc46-i486bs0): https://archlinux32.org
[18:49:45] <buildmaster> i486/xorg-server is broken (says nlopc46-i486bs0): https://archlinux32.org
[19:03:50] <buildmaster> dirty! girls, my database - so dirty :-(
[19:03:50] * buildmaster goes insane.
[19:07:13] <nit-picker> dependencies of xorg-server-xephyr-1.20.5-2.4-i486.pkg.tar.xz (built on 2019-09-14) differ between the package and our database
[19:11:11] -!- thePiGrepper has quit [Ping timeout: 258 seconds]
[19:12:05] -!- thePiGrepper has joined #archlinux32
[19:14:16] <nit-picker> dependencies of xorg-server-devel-1.20.5-2.4-i486.pkg.tar.xz (built on 2019-09-14) differ between the package and our database
[19:36:10] <nit-picker> dependencies of xorg-server-1.20.5-2.4-i486.pkg.tar.xz (built on 2019-09-14) differ between the package and our database
[20:13:45] -!- davor_ has joined #archlinux32
[20:14:19] -!- davor has quit [Ping timeout: 272 seconds]
[20:14:19] davor_ is now known as davor
[20:55:07] <tyzoid> abaumann: I install packages on the ramdisk all the time, it essentially needs you to install the archlinux32-keyring package and do a `pacman-key --populate archlinux32`
[22:36:24] * buildmaster resumes sanity.
[22:37:20] <girls> someone / something removes packages from the mysql database without removing them from the master mirror - that's also what nit-picker complained about: the dependencies differed, because the package in the database *was gone*
[22:41:08] <tyzoid> Was the upstream package split?
[22:41:25] <girls> I doubt it, but let me recheck
[22:41:39] <tyzoid> It could be a false positive split detection
[22:41:40] <girls> this also only happened on i486, currently
[22:41:51] <girls> last time it happened on multiple architectures, but for different packages
[22:42:08] <tyzoid> Do we have transaction logs?
[22:42:15] <tyzoid> perhaps we could identify the query that caused the removal
[22:42:24] <girls> we do not log the queries
[22:42:32] <girls> (besides what mariadb does for replication)
[22:42:39] <girls> but we have a log of run shell commands
[22:42:55] <tyzoid> Is it transactional replication? or row-replication?
[22:42:56] <girls> no, split-status did not change
[22:43:15] <tyzoid> transaction replication would effectively log queries
[22:43:17] <girls> tyzoid: sry, I don't know the difference
[22:43:27] <girls> there is a "binary log" which is used
[22:44:09] <tyzoid> Regardless of the method, the log could likely be used to find when the deletion occurred and correlate it to a script process
[22:44:26] <girls> right, we have hourly dumps of the database
[22:44:41] <tyzoid> The transaction/replication log would have much finer detail
[22:45:01] <tyzoid> likely would be able to correlate to a within a few seconds
[22:45:51] <girls> /var/lib/mysql/mysql-bin.004577 and alike seem to containt the content
[22:47:58] <tyzoid> mysqlbinlog is the program to read them: https://dev.mysql.com
[22:47:59] <phrik> Title: MySQL :: MySQL 5.7 Reference Manual :: 4.6.7 mysqlbinlog — Utility for Processing Binary Log Files (at dev.mysql.com)
[22:49:38] <girls> mysqlbinlog: unknown variable 'default-character-set=utf8mb4'
[22:49:39] <girls> hmmm
[22:55:28] <tyzoid> weird
[22:55:56] <girls> it was removed between 18:23:38 and 19:23:18
[22:55:59] <girls> says bisect
[22:56:12] <tyzoid> Hmm
[22:56:20] <tyzoid> that's a lot less granularity that I would expect
[22:56:41] <girls> well, hourly dumps only give hour granularity ;-)
[22:56:51] <tyzoid> Ah, I thought that was the binlog
[22:56:59] <tyzoid> try calling the binlog program with `--no-defaults`
[22:57:03] <tyzoid> see if that helps
[22:57:38] <girls> yes
[22:57:41] <girls> now it exits with 0
[22:57:57] <tyzoid> That... does not sound very enthusiastic
[22:58:01] <tyzoid> lol
[22:58:08] <girls> well, it does not error
[22:58:16] <girls> but I don't understand how to use this tool
[22:58:29] <girls> maybe, you can try yourself on the buildmaster?
[22:59:01] <tyzoid> Sure, let me see if I can access it via proxy
[22:59:17] <girls> ah, right, you're at work
[23:00:09] <tyzoid> Nothing a little xterm.js and SSH port forwarding can't fix
[23:00:56] <girls> there was not much running between 18:23:38 (last good dump) and 19:03:50 (buildmaster goes insane)
[23:02:14] <tyzoid> damn, binlog uses two digits for year :/
[23:02:33] <tyzoid> Prepare for Y2.1k bugs
[23:02:52] <tyzoid> 18:23:33 and 19:23:18 are times UTC?
[23:03:47] <girls> probably UTC+1
[23:03:49] <girls> let me check
[23:04:02] <girls> yes, UTC+1
[23:04:06] <tyzoid> the mysql logs are in localtime, which is UTC+1 for Europe/Berlin
[23:04:11] <tyzoid> so that makes matching easier
[23:04:41] <girls> 18:23:33 and 19:03:50 in UTC+1, right
[23:05:26] <tyzoid> What fun, the binlogs rotated at 20:23 and 21:23 lol
[23:05:37] <tyzoid> matches up exactly with our hour bisect interval
[23:06:00] <girls> probably rotated due to the running mysqldump?
[23:06:09] <tyzoid> possibly
[23:06:24] <tyzoid> whatever the reason, makes it nicer for me only having to look at one log file
[23:06:43] <girls> look for xorg-server being deleted for architecture 3
[23:07:00] <girls> binary_packages.id=446293
[23:07:09] <girls> binary_packages_in_repository.id=774468
[23:08:26] <tyzoid> ewww all the data is in base64
[23:09:07] <tyzoid> also I'm not even seeing those tables
[23:09:30] <girls> sry, typo: it's `binary_packages_in_repositories`
[23:09:33] <tyzoid> I'm only seeing the ssh_logs and build_slaves tables
[23:09:46] <girls> hmmm
[23:10:36] <tyzoid> Here's a question: the buildmaster is the master (write) database for the package data, right?
[23:10:43] <girls> right
[23:10:46] <tyzoid> so the replication logs would only appear on the server it's replicating to
[23:11:04] <girls> I thought, the slaves pull the replication logs from the master
[23:11:06] <girls> but I might be wrong
[23:11:23] <girls> in the end, I understand more about nonlinear optics than about relational database replication ;-)
[23:11:28] <tyzoid> When mysql does replication, it writes a log of changes made that were caused by upstream
[23:11:35] <girls> ok
[23:12:11] <tyzoid> simplified statement, ofc, because I don't fully understand it either. I've not set one up before, but I've worked with replicated databases a little before
[23:12:36] <tyzoid> But we should look for the binlogs on a server that's a slave to this data
[23:14:53] <girls> looks, like the relay bin logs are removed after 2 hours or so on my replication slave
[23:14:54] <girls> :-/
[23:15:04] <tyzoid> :/
[23:15:17] <tyzoid> Well, now we know what we're looking for
[23:15:43] <tyzoid> so we can try to triage based on the one hour block, and then pull data the next time the package disappears
[23:17:25] <girls> hah, I think, I found the faulty command
[23:17:32] <girls> return-assignment xorg-server bc3ecf1b75be0bf89d7d528a8e7d26c29cfcfaf8 f445e64d575082235d5ab21c6a0f890c90af620d extra i486 ERROR
[23:17:38] <girls> was run in the respective window
[23:18:42] <tyzoid> Why would a return of error cause a deletion though?
[23:18:51] <tyzoid> I thought a build error was a common event
[23:19:26] <girls> maybe, the package was marked for deletion beforehand
[23:19:30] <girls> (blacklisting)
[23:21:00] <girls> hmm
[23:21:06] <girls> the queries look all ok
[23:21:22] <girls> maybe we should really just log *all* queries >:-)
[23:21:44] <tyzoid> BTW, here's a full list of tables that hit the replication log in that one hour: ssh_log, build_slaves, fluxbb_online, fluxbb_search_cache, fluxbb_topics, command_log, ps, fluxbb_users,
[23:24:22] <tyzoid> girls: The binlog looks like it does log pretty much all queries
[23:25:18] <tyzoid> Perhaps we should increase the amount of time the replication logs are kept?
[23:25:37] <girls> yeah, probably
[23:25:55] <tyzoid> I imagine it was probably set to only two hours because those logs are pretty large in size
[23:49:08] <girls> I'm afraid, I'm overdue