Hi Paolo,
The performance compare and bisect tools will both deal with stats and there is a library that provides some few fundamental stats functions:
https://gitee.com/wu_fengguang/lkp-tests/blob/master/lib/stats.rb
You may check and select the parts that can be referenced or directly reused when doing performance regression detection and bisection.
Xueliang wrote a simple bisect tool that works for build errors:
https://gitee.com/wu_fengguang/compass-ci/blob/master/sbin/git-bisect
Which may be referenced when doing performance bisect tool.
The BJTU students in CC list are interested in bisect. Hopefully you can cooperate on the work.
Thanks,
Fengguang
---
> Il giorno 9 feb 2021, alle ore 12:50, wufengguang <wufengguang(a)huawei.com> ha scritto:
>
> Hi Anmar,
>
> Sorry for top posting -- it's outlook.
>
> lkp-tests does have the compare scripts
>
> https://gitee.com/wu_fengguang/lkp-tests/blob/master/sbin/compare
> https://gitee.com/wu_fengguang/lkp-tests/blob/master/sbin/ncompare
>
> However they work on data from local files, which does not scale.
> So we created
>
> https://gitee.com/openeuler/compass-ci/blob/master/sbin/compare
>
> to work on data from ElasticSearch database.
>
> Some compare functions can be reused as base for detect/bisecting performance regression. However we still need add more functions to actually detect regression and bisect good/bad state. That is the current state.
>
Hi Fengguang,
how are you?
We are about to start implementing our tool for detecting regressions. In this respect, I'd like to make sure I've got the following important point right. So let me sum up the situation with my words and ask you whether I'm right.
Functionalities for detecting regression are not mature yet in Compass-CI, whereas 01.org does report performance regressions. So, compass-ci and 01.org are both based on LKP, but 01.org guys have already implemented their own code for regression detection and reporting. This code has not to do with the compare functionalities your are developing.
Am I correct?
Thanks,
Paolo
> Thanks,
> Fengguang
> ---
>
> Hello Fengguang,
>
> I hope this email finds you well. We dug into the regression detection tool and discovered things like lib/compare and reports such as [1] clearly showing that LKP is able to do some regression analysis.
>
> TBH, I'm a little confused as to what the regression detection tool we discussed is meant to do given that the functionality is already present in LKP. I recall you mentioning that CompassCI doesn't do regression detection so I might have misunderstood your statement or something got lost in translation. Our intention is to create a benchmark data analysis and regression tool that can be used by both CompassCI and LKFT.
>
> Let me know what you think please. We can always jump on a call to sort this out if need be.
>
> --
> anmar
>
>
> [1] https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/V725VQJIASSE2N…
Hi Anmar,
Sorry for top posting -- it's outlook.
lkp-tests does have the compare scripts
https://gitee.com/wu_fengguang/lkp-tests/blob/master/sbin/comparehttps://gitee.com/wu_fengguang/lkp-tests/blob/master/sbin/ncompare
However they work on data from local files, which does not scale.
So we created
https://gitee.com/openeuler/compass-ci/blob/master/sbin/compare
to work on data from ElasticSearch database.
Some compare functions can be reused as base for detect/bisecting performance regression. However we still need add more functions to actually detect regression and bisect good/bad state. That is the current state.
Thanks,
Fengguang
---
Hello Fengguang,
I hope this email finds you well. We dug into the regression detection tool and discovered things like lib/compare and reports such as [1] clearly showing that LKP is able to do some regression analysis.
TBH, I'm a little confused as to what the regression detection tool we discussed is meant to do given that the functionality is already present in LKP. I recall you mentioning that CompassCI doesn't do regression detection so I might have misunderstood your statement or something got lost in translation. Our intention is to create a benchmark data analysis and regression tool that can be used by both CompassCI and LKFT.
Let me know what you think please. We can always jump on a call to sort this out if need be.
--
anmar
[1] https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/V725VQJIASSE2N…
Hello Fengguang,
I hope this email finds you well. We dug into the regression
detection tool and discovered things like lib/compare and reports such
as [1] clearly showing that LKP is able to do some regression
analysis.
TBH, I'm a little confused as to what the regression detection tool we
discussed is meant to do given that the functionality is already
present in LKP. I recall you mentioning that CompassCI doesn't do
regression detection so I might have misunderstood your statement or
something got lost in translation. Our intention is to create a
benchmark data analysis and regression tool that can be used by both
CompassCI and LKFT.
Let me know what you think please. We can always jump on a call to
sort this out if need be.
--
anmar
[1] https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/V725VQJIASSE2N…
Hello anmar,
It's nice to meet you too. I have some questions and need your help.
1. Should i install OpenEuler system in ARM or hard disk?
2. Could you provide LKFT deploy script?
3. Our server maybe power off in some special cases, how long do i have to tell you in advance?
Thanks,
Yinsi
liuyinsi(a)163.com
From: wufengguang
Date: 2021-01-13 18:50
To: liuyinsi(a)163.com
Subject: Fwd: Remote LKFT dispatcher
发件人: Anmar Oueja [mailto:anmar.oueja@linaro.org]
发送时间: 2021年1月12日 10:20
收件人: wufengguang <wufengguang(a)huawei.com>
抄送: lkq-dev(a)op-lists.linaro.org; Jammy Zhou <jammy.zhou(a)linaro.org>; Jonathan Cameron <jonathan.cameron(a)huawei.com>; Guohanjun (Hanjun Guo) <guohanjun(a)huawei.com>
主题: Remote LKFT dispatcher
Hello Fengguang,
I hope this email finds you well. I discussed the details of using one of your servers as a remote instance for KVM Qemu DUTs and it seems pretty straightforward. Here is a brief proposal to things rolling. Feedback is welcome.
Goals
--------
* Expand LKFT's Aarch64 test capacity using KVM Qemu using Huawei server as a host system.
* Provide a real workload to test of OpenEuler OS by using it as a host OS to host the Qemu KVM instances
Deliverables
-----------------
* OpenEuler to setup and configure a dedicated Huawei server for remote access
* OpenEuler to Install and configure its OS on the machine
* OpenEuler to grant specific Linaro individuals access to the server with Sudo permissions (dispatcher container requires root access)
* Linaro will install and configure a remote dispatcher that will be registered in the LKFT master instance
* Linaro will initially use the server in its staging instance to allow issues be identified and resolved before moving it to the production instance
* Linaro to move the server to LKFT's production instance
Assumptions
-----------------
* OpenEuler community will administer the server and maintain it including keeping up with the dispatcher monthly updates (as part of LAVA's monthly releases)
* Adequate storage and memory are provided to maximise the number of simultaneous Qemu instances
* Network bandwidth is fast enough to download all the images and test artifacts needed to run the tests and report the results in a timely fashion
Risks
-------
* Once the server is moved to the production instance of LKFT, uptime commitments from the OpenEuler project will be required. This can be discussed at a later date.
Thanks,
anmar
Hi Anmar,
Got it, it looks a good plan！
Yinsi (CCed) from our team will help set up LKFT in our environment.
She’ll likely need your team’s kind support in future. :)
Thanks,
Fengguang
-----------------------------------
Hello Fengguang,
I hope this email finds you well. I discussed the details of using one of your servers as a remote instance for KVM Qemu DUTs and it seems pretty straightforward. Here is a brief proposal to things rolling. Feedback is welcome.
Goals
--------
* Expand LKFT's Aarch64 test capacity using KVM Qemu using Huawei server as a host system.
* Provide a real workload to test of OpenEuler OS by using it as a host OS to host the Qemu KVM instances
Deliverables
-----------------
* OpenEuler to setup and configure a dedicated Huawei server for remote access
* OpenEuler to Install and configure its OS on the machine
* OpenEuler to grant specific Linaro individuals access to the server with Sudo permissions (dispatcher container requires root access)
* Linaro will install and configure a remote dispatcher that will be registered in the LKFT master instance
* Linaro will initially use the server in its staging instance to allow issues be identified and resolved before moving it to the production instance
* Linaro to move the server to LKFT's production instance
Assumptions
-----------------
* OpenEuler community will administer the server and maintain it including keeping up with the dispatcher monthly updates (as part of LAVA's monthly releases)
* Adequate storage and memory are provided to maximise the number of simultaneous Qemu instances
* Network bandwidth is fast enough to download all the images and test artifacts needed to run the tests and report the results in a timely fashion
Risks
-------
* Once the server is moved to the production instance of LKFT, uptime commitments from the OpenEuler project will be required. This can be discussed at a later date.
Thanks,
anmar
Hello Fengguang,
I hope this email finds you well. I discussed the details of using one of
your servers as a remote instance for KVM Qemu DUTs and it seems pretty
straightforward. Here is a brief proposal to things rolling. Feedback is
welcome.
Goals
--------
* Expand LKFT's Aarch64 test capacity using KVM Qemu using Huawei server as
a host system.
* Provide a real workload to test of OpenEuler OS by using it as a host OS
to host the Qemu KVM instances
Deliverables
-----------------
* OpenEuler to setup and configure a dedicated Huawei server for remote
access
* OpenEuler to Install and configure its OS on the machine
* OpenEuler to grant specific Linaro individuals access to the server with
Sudo permissions (dispatcher container requires root access)
* Linaro will install and configure a remote dispatcher that will be
registered in the LKFT master instance
* Linaro will initially use the server in its staging instance to allow
issues be identified and resolved before moving it to the production
instance
* Linaro to move the server to LKFT's production instance
Assumptions
-----------------
* OpenEuler community will administer the server and maintain it including
keeping up with the dispatcher monthly updates (as part of LAVA's monthly
releases)
* Adequate storage and memory are provided to maximise the number of
simultaneous Qemu instances
* Network bandwidth is fast enough to download all the images and test
artifacts needed to run the tests and report the results in a timely fashion
Risks
-------
* Once the server is moved to the production instance of LKFT, uptime
commitments from the OpenEuler project will be required. This can be
discussed at a later date.
Thanks,
anmar
Hi,
The meeting minutes are summarized below. Thanks for attending.
MINUTES:
Vendor remote lab is interesting to openEuler for wide hardware coverage.
-
Leverage existing Linaro LKFT Lab + remote lab
-
e.g, NXP remote lab (shared by NXP, Linaro, KernelCI)
-
https://lavalab.nxp.com/
-
CompassCI also has plan for remote lab support
CompassCI also has the capability for bisecting support (the algorithm can
be universal to both kernel and user space applications). LKP itself
doesn’t have the bisecting capability. TuxSuite is focusing mainly on the
kernel part.
Paolo: is it functional or performance bisecting in CompassCI?
Fengguang: currently functional, performance support will be added later
CompassCI starts from upstream, and then downstream (so not only downstream
kernel).
Arch Linux package build is used by CompassCI, so it is easy to add
different kernel builds.
Scale testing with QEMU (e.g, run openEuler on Kunpeng 920, and run
multiple QEMU instances on openEuler), and it can be deployed via remote
lab. Fengguang: it is nice.
17+ machines in CompassCI lab now. Maybe start with 1 machine for
prototyping (the dispatcher can run on the same machine for QEMU case).
Anmar: the dispatcher is fully dockerized now, there should be no big issue
to run on openEuler.
Paolo: modularization can be helpful to merge the benefits of both
Paolo: Is the part for performance bisecting ready in CompassCI?
Fengguang: not yet
Paolo: maybe we can work together to have some common solution to avoid
reinventing the wheel. It’s under design for LKFT so it’s a good place for
us to collaborate.
Fengguang: Focus is on build now then the benchmark testing.
Anmar: the Linaro team and Fengguang to connect later on to make sure the
tool is useful for both CI systems.
Questions and Answers:
-
Q. Can LKP run without having to install packages or tests, only the job
definitions (scripts) ? If we can run tests without having to install the
pre-req every time, that will speed up testing if we use a rootfs that has
all its pre-req installed.
-
A. Compass CI has cpio archives it uses to overlay the tests
dependencies to speed the test runs. These overlays are pre-built and
available to the Compass-CI system
-
Q. We want to get a better understanding of the compare functionality
-
A. compare fetches different results and presents them. Nothing more.
-
Q. What are the plans for alignment with upstream LKP? By examining the
https://github.com/fengguang/lkp-tests and comparing it to
https://github.com/intel/lkp-tests.git, it seems the trees started
diverging in March 2020.
-
A. Fengguang’s LKP-tests repo is periodically re-based on the upstream
LKP-tests repo every couple of months. There is effort to push some of the
changes back upstream but been minimal so far. Most of the tests run on
Arm64 so from a testing perspective, Arm64 is well represented. It’s also
not clear if LKP will accept many patches at the moment. However, a
discussion needs to happen with the upstream at some point in time.
-
Q. How difficult is it to modularise the following LKP components:
-
Monitors
-
Tests
-
Test-plans
-
Email reports
-
Harmonizing output
-
Functional
-
Benchmark
-
Regression detection
-
A. LKP is already modularized and functions as a pipeline.
Regards,
Jammy
On Wed, 9 Dec 2020 at 17:03, Jammy Zhou via Lkq-dev <
lkq-dev(a)op-lists.linaro.org> wrote:
> Hi All,
>
> We're going to have some follow up discussion about Compass CI and LKFT.
> Welcome to join.
>
> @Anmar, please help share the detailed agenda to the list when available.
>
> ------
> Topic: CompassCI/LKFT follow up discussion
> Time: Dec 10, 2020 09:00 PM Hong Kong SAR
>
> Join Zoom Meeting
> https://linaro-org.zoom.us/j/97232094091
>
> Meeting ID: 972 3209 4091
> One tap mobile
> +16699009128,,97232094091# US (San Jose)
> +12532158782,,97232094091# US (Tacoma)
>
> Dial by your location
> +1 669 900 9128 US (San Jose)
> +1 253 215 8782 US (Tacoma)
> +1 301 715 8592 US (Washington D.C)
> +1 312 626 6799 US (Chicago)
> +1 346 248 7799 US (Houston)
> +1 646 558 8656 US (New York)
> 888 788 0099 US Toll-free
> 877 853 5247 US Toll-free
> Meeting ID: 972 3209 4091
> Find your local number: https://linaro-org.zoom.us/u/ad2qaJfAiY
>
> Regards,
> Jammy
> --
> Lkq-dev mailing list
> Lkq-dev(a)op-lists.linaro.org
> https://op-lists.linaro.org/mailman/listinfo/lkq-dev
>
Hello Fengguang,
I hope this email finds you well. We have a few questions we would
appreciate your insight on. We can discuss the answers on this email thread
or at the tail end of our call tomorrow.
- Can LKP run without having to install packages or tests, only the job
definitions (scripts) ? If we can run tests without having to install the
pre-req every time, that will speed up testing if we use a rootfs that has
all its pre-req installed.
- We want to get a better understanding of the benchmark parser
including how benchmark regression are detected
- We want to get a better understanding of the compare functionality
- What are the plans for alignment with upstream LKP? By examining the
https://github.com/fengguang/lkp-tests and comparing it to
https://github.com/intel/lkp-tests.git, it seems the trees started
diverging in March 2020.
- Which tests are Intel specific? Are there plans for porting them to
Arm64?
- How difficult is it to modularise the following LKP components:
- Monitors
- Tests
- Test-plans
- Email reports
- Harmonizing output
- Functional
- Benchmark
- Regression detection
We look forward to talking to you tomorrow.
Cheers!
anmar
Hi All,
We're going to have some follow up discussion about Compass CI and LKFT.
Welcome to join.
@Anmar, please help share the detailed agenda to the list when available.
------
Topic: CompassCI/LKFT follow up discussion
Time: Dec 10, 2020 09:00 PM Hong Kong SAR
Join Zoom Meeting
https://linaro-org.zoom.us/j/97232094091
Meeting ID: 972 3209 4091
One tap mobile
+16699009128,,97232094091# US (San Jose)
+12532158782,,97232094091# US (Tacoma)
Dial by your location
+1 669 900 9128 US (San Jose)
+1 253 215 8782 US (Tacoma)
+1 301 715 8592 US (Washington D.C)
+1 312 626 6799 US (Chicago)
+1 346 248 7799 US (Houston)
+1 646 558 8656 US (New York)
888 788 0099 US Toll-free
877 853 5247 US Toll-free
Meeting ID: 972 3209 4091
Find your local number: https://linaro-org.zoom.us/u/ad2qaJfAiY
Regards,
Jammy
Hi Anmar,
> Can you point me at the OpenEuler kernel trees and the branches you guys use?
The openEuler kernel tree is
https://gitee.com/openeuler/kernel
The current active branches are
kernel-4.19
openEuler-20.09
Thanks,
Fengguang
