Build timers #5

New Issue

kreyren · 2021-11-22T22:56:57+01:00

kreyren commented

2021-11-22 22:56:57 +01:00

Proposing the integration of build timers meaning outputting how long the build is expected to take and how long it took to:

a) Preventing wasting of time and resources
I tried to build guix on my weak notebook with 1 core that after 47 hours failed with insufficient memory where build timers would contribute to preventing this situation

b) Enable time management
If i am doing something and i know that it's going to take 2 hours to finish then i can go and do something else instead of babysitting the system until it finished without knowing whether it's going to finish in the next minute or next hour.

c) Enable resource management
If the build is going to take e.g. 12 hours to build then knowing it in advance would help with resource management like setting up off-loading to avoid being in a situation where the build is taking 6 hours and it being difficult to make a decision to kill it thinking it might finish any second.

d) Making the build process more robust for mission critical environment

If we have a build that is projected to take 2 min and it's taking 3 minutes then it's safe to assume that something is going wrong like infinite loop during build or other halting scenario.

Proposed integration

Integrate timers that measure how long the build took and then report that time with relevant telemetry to specified logging.

For the start we should only collect value of ( max cpu frequency / build time) * cpu threads which should give us sufficient average for the build time but the following incluence the calculations:

System noise during the build
CPU overheating -> TEMP_MAX and TEMP_CRITICAL trigger fail-safe that lowers the cpu efficiency until it reaches safer temp
RAM speed
Storage device's read+write speed

Privacy: This feature should be opt-in to enforce privacy and in a way that allows the user to view what exactly is being sent for processing.

Profiling: New packages should include requirement for profiling to report how long it took to build the package. The package has to be built multiple times prior to merge so this shoudn't add up any overhead.

Decentralization: Should be implemented in a decentralized-way to ensure that the process is not dependent on a centralized server/database

Credit: Gentoo Linux's equery

Proposing the integration of build timers meaning outputting how long the build is expected to take and how long it took to: **a) Preventing wasting of time and resources** I tried to build guix on my weak notebook with 1 core that after 47 hours failed with insufficient memory where build timers would contribute to preventing this situation **b) Enable time management** If i am doing something and i know that it's going to take 2 hours to finish then i can go and do something else instead of babysitting the system until it finished without knowing whether it's going to finish in the next minute or next hour. **c) Enable resource management** If the build is going to take e.g. 12 hours to build then knowing it in advance would help with resource management like setting up off-loading to avoid being in a situation where the build is taking 6 hours and it being difficult to make a decision to kill it thinking it might finish any second. **d) Making the build process more robust for mission critical environment** If we have a build that is projected to take 2 min and it's taking 3 minutes then it's safe to assume that something is going wrong like infinite loop during build or other halting scenario. ### Proposed integration Integrate timers that measure how long the build took and then report that time with relevant telemetry to specified logging. For the start we should only collect value of `( max cpu frequency / build time) * cpu threads` which should give us sufficient average for the build time but the following incluence the calculations: 1. System noise during the build 2. CPU overheating -> TEMP_MAX and TEMP_CRITICAL trigger fail-safe that lowers the cpu efficiency until it reaches safer temp 3. RAM speed 4. Storage device's read+write speed **Privacy:** This feature should be opt-in to enforce privacy and in a way that allows the user to view what exactly is being sent for processing. **Profiling:** New packages should include requirement for profiling to report how long it took to build the package. The package has to be built multiple times prior to merge so this shoudn't add up any overhead. **Decentralization:** Should be implemented in a decentralized-way to ensure that the process is not dependent on a centralized server/database --- Credit: Gentoo Linux's equery

kreyren commented

2021-11-23 07:29:27 +01:00

Theory: The Pokemon Battle Analogy

In terms of mathemetically determining when the build will be finished using the (max cpu frequency * cpu threads) we can look on the issue as pokemon battle:

Health Points -> The complexity of the package
Damage per second ("DPS") of the CPU -> value of (max cpu frequency * cpu threads)

To then determine that package with calculated 500 HP will need either a CPU that deals 100HP/s to finish in 5 sec or CPU that deals 50HP/s to finish in 10 sec.

Observed issues with this theory are determining the package's complexity while minding the possible interference from system noise in the damage per second of the CPU.

This way we can theoretically calculate a very accurate measurement for how long it will take to build across multiple systems.

Theory for real-time measurements:
In situation where package has 2000 HP the package building at 5 DPS for 1 min which would take 400s to finish at this rate (2000 HP/4 DPS=400), but if e.g. the system turns on boost clock that changes the CPU to deal 8 DPS then that would take down the overall build time to [2000HP-(60sec*5 HP)]/8 HP = 212.5sec to finish

### Theory: The Pokemon Battle Analogy In terms of mathemetically determining when the build will be finished using the `(max cpu frequency * cpu threads)` we can look on the issue as pokemon battle: 1. Health Points -> The complexity of the package 2. Damage per second ("DPS") of the CPU -> value of `(max cpu frequency * cpu threads)` To then determine that package with calculated 500 HP will need either a CPU that deals 100HP/s to finish in 5 sec or CPU that deals 50HP/s to finish in 10 sec. Observed issues with this theory are determining the package's complexity while minding the possible interference from system noise in the damage per second of the CPU. This way we can theoretically calculate a very accurate measurement for how long it will take to build across multiple systems. **Theory for real-time measurements:** In situation where package has 2000 HP the package building at 5 DPS for 1 min which would take 400s to finish at this rate (2000 HP/4 DPS=400), but if e.g. the system turns on boost clock that changes the CPU to deal 8 DPS then that would take down the overall build time to \[2000HP-(60sec*5 HP)]/8 HP = 212.5sec to finish

Sign in to join this conversation.

No Label

No Milestone

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: guix.next/GUIX.next#5