Hi, all
According to the discussion, the PR generated by the openEuler docker image has been provided.
It support the build of puppet and slaves docker, the base docker can pull from the openEuler web.
Add OpenEuler 22.03 support
PR address: https://github.com/apache/bigtop/pull/1051
Issue address: https://issues.apache.org/jira/browse/BIGTOP-3873
On this, we can upload the patch files of components one by one, for support openEuler OS.
Please check it out and give suggestions, thx.
发件人: Yangjian (nt)
发送时间: 2022年9月26日 19:19
收件人: bdds-dev(a)op-lists.linaro.org
抄送: chenqiang (N) <mac.chenqiang(a)huawei.com>; Guodong Xu <guodong.xu(a)linaro.org>; Luoliyuan <luoliyuan(a)huawei.com>
主题: [Bdds-dev] Some problems with bigtop are adapted to openEuler system, pls help to reply, tks.
English:
1. Where can I get the component installation and deployment information during the smoke-test?
Such as include component startup options and commands? What options do the test components need to configure?
2. Is the smoke-test using a single machine or a cluster?
3. Smoke-test test cases without Ranger 2.2.0 components.
4. The gpdb 5.28.5 component does not support python3 when compiled.
5. ycsb 0.17.0 component smoke test error log:
File "/usr/lib/ycsb/bin/ycsb", line 228
except subprocess.CalledProcessError, err:
^
SyntaxError: invalid syntax
6.We solved the problem and raised a PR:
https://github.com/apache/bigtop/pull/1025
Chinese:
1、冒烟测试过程中组件安装部署信息在哪儿可以获取到?比如包括组件启动选项和命令是什么?测试组件需要配置哪些选项?
2、冒烟测试是用单机还是集群?
3、冒烟测试没有ranger组件的测试用例
4、gpdb 5.28.5 组件编译时不支持python3
5、ycsb 0.17.0 组件冒烟测试报错:
File "/usr/lib/ycsb/bin/ycsb", line 228
except subprocess.CalledProcessError, err:
^
SyntaxError: invalid syntax
6、我们解决问题提了一个PR:
https://github.com/apache/bigtop/pull/1025
Best Regards
English:
1. Where can I get the component installation and deployment information during the smoke-test?
Such as include component startup options and commands? What options do the test components need to configure?
2. Is the smoke-test using a single machine or a cluster?
3. Smoke-test test cases without Ranger 2.2.0 components.
4. The gpdb 5.28.5 component does not support python3 when compiled.
5. ycsb 0.17.0 component smoke test error log:
File "/usr/lib/ycsb/bin/ycsb", line 228
except subprocess.CalledProcessError, err:
^
SyntaxError: invalid syntax
6.We solved the problem and raised a PR:
https://github.com/apache/bigtop/pull/1025
Chinese:
1、冒烟测试过程中组件安装部署信息在哪儿可以获取到?比如包括组件启动选项和命令是什么?测试组件需要配置哪些选项?
2、冒烟测试是用单机还是集群?
3、冒烟测试没有ranger组件的测试用例
4、gpdb 5.28.5 组件编译时不支持python3
5、ycsb 0.17.0 组件冒烟测试报错:
File "/usr/lib/ycsb/bin/ycsb", line 228
except subprocess.CalledProcessError, err:
^
SyntaxError: invalid syntax
6、我们解决问题提了一个PR:
https://github.com/apache/bigtop/pull/1025
Best Regards
Hi ALL:
1. Which version of Bigtop should be based on to support openEuler OS ?(3.1.1 or 3.2.2 ?)
Is there a development plan and roadmap for this release?
2. The current stable version of openEuler OS is 22.03, and some system software version information is as follows.
What should I do if it is inconsistent with your planned version?
For example, the higher python version is 3.9.9, and hadoop compilation requires python 2.7 version.
[cid:image001.png@01D8B661.AA6A3640]
---------- Forwarded message ---------
From: Kengo Seki <sekikn(a)apache.org>
Date: Mon, Oct 25, 2021 at 11:32 AM
Subject: [ANNOUNCE] Apache Bigtop 3.0.0 released
To: <announce(a)apache.org>, <user(a)bigtop.apache.org>, dev <
dev(a)bigtop.apache.org>
On behalf of the Apache Bigtop team, I'd love to announce the general
availability of the Bigtop 3.0.0 release.
The release is available here:
https://bigtop.apache.org/download.html#releases
A few highlights of this release include:
* Major components such as Hadoop, Hive and Spark are finally upgraded to
3.x.
* Other components are also upgraded so that they are compatible
with the above ones and users can enjoy their enhancement.
* Fedora 33 and Ubuntu 20.04 are newly added as the supported distros.
* In addition to x86_64 and aarch64, ppc64le is also supported,
which was not in Bigtop 1.5.0.
With Bigtop 3.0.0 the community continues to deliver the most advanced
big data stack to date. More details about 3.0.0 release are here:
https://bigtop.apache.org/release-notes.html
Deploying Bigtop is easy: grab the repo/list file for your favorite
Linux distribution:
https://www.apache.org/dyn/closer.lua/bigtop/bigtop-3.0.0/repos/
and you'll be running your very own bigdata cluster in no time!
We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at:
https://bigtop.apache.org
Lastly, I want to emphasize that this is a collaborative work done by
project contributors and other communities,
who continue to devote time to make Bigtop a better software. Thank
you all for making this release possible!
Regards,
Kengo Seki, Bigtop 3.0.0 Release Manager
--
IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Hi All,
Thanks for joining the call today. Below are some notes from this meeting.
*Attendees*:
Gugang, Shenshu, Xiaoqi Ma, Jian Yang, Yuqi, Jun He, Ganesh, Jammy, Elsie
*Notes*:
- Current BigTop release is v1.5, now moving to 3.0 to integrate Hadoop
v3.x, and Ambari mpack support is in progress
- ApacheCon 2021 is coming (Sep. 21~23), and some announcement is
planned about BigTop release v3.0
- People who use HDP and CDH start looking at BigTop
- Future plans and ideas for BigTop (need to prioritize)
- Kubernetes support
- More components to be added like Apache Beam, Arrow, Ranger, etc
- Looking forward to feedback from Huawei and other vendors about BigTop
and other Big Data and Data Science projects
- JIRA link for BigTop v3.0 release
https://issues.apache.org/jira/browse/BIGTOP-3471
- BigTop mailing list http://bigtop.apache.org/mail-lists.html
- Shenshu shared some requirements (by comparing to Ambari+HDP)
- deploy all components by Ambari
- support openEuler/Kylin/UOS operating systems
- upgrade some components to new versions such as
Hadoop/Hive/Hbase/etc
- add Pig/Storm/Ranger/Atlas/Knox/Accumulo/Druid/Impala/Kudu
- provide a install package (tar.gz) that contains all the rpms/debs
- every component should release a tar.gz package
- Most components will be upgraded for BigTop v3.0, and the missing
components can be interesting to add
- Solr will be upgraded to 8.7 for BigTop v3.0
- Gang: current CI has only x86 and CentOS for components like Hadoop,
Spark, Hbase in the BigTop community, doesn’t have Aarch64
- Jun: the work for v3.0 is still ongoing, some CI support is
disabled temporarily, for the final release, x86/Aarch64/PowerPC will all
be supported
- Gang: Ambari mpack can only detect several components now, which is
quite limited
- Yuqi: Ambari will be upgraded to 2.7.5, there will be some work for
mpack, we can do together if there is some special requirements (the core
components have high priority)
- Hadoop 3.3 has official support for AArch64
- Impala and Kudu will be required (please raise a ticket in the JIRA
link for BIGTOP-3471), similar for other requirements
- For BigTop, it is hard to have a single tar file for all components.
Encourage to participate in the BigTop community (e.g, mailing list) for
related discussions
- The BigTop community is open to add more distro support (like
openEuler). Is there some docker image for openEuler?
- Yes, docker image can be found in the docker_img folder of
openEuler repo, like
https://repo.openeuler.org/openEuler-20.03-LTS-SP1/docker_img/
Regards,
Jammy
On Wed, 26 May 2021 at 20:43, Jammy Zhou via Bdds-dev <
bdds-dev(a)op-lists.linaro.org> wrote:
> Hi All,
>
> With recent changes in the industry, BigTop is getting more momentum as an
> open source distro for Big Data components like Hadoop, Spark and HBase.
>
> We're going to host a meeting to discuss how we can work together in the
> community to enhance it and resolve the gaps/issues. For example:
>
> - What components are still missed?
> - What component versions should be used?
> - Any problems for using with Ambari?
>
> Please check the meeting info below, and welcome to join.
> ============
> Topic: BigTop open discussion
> Time: May 28, 2021 10:00 AM CST (UTC+8)
>
> Join Zoom Meeting
> https://linaro-org.zoom.us/j/98394347833
>
> Meeting ID: 983 9434 7833
> One tap mobile
> +16699009128,,98394347833# US (San Jose)
> +12532158782,,98394347833# US (Tacoma)
>
> Dial by your location
> +1 669 900 9128 US (San Jose)
> +1 253 215 8782 US (Tacoma)
> +1 301 715 8592 US (Washington DC)
> +1 312 626 6799 US (Chicago)
> +1 346 248 7799 US (Houston)
> +1 646 558 8656 US (New York)
> 888 788 0099 US Toll-free
> 877 853 5247 US Toll-free
> Meeting ID: 983 9434 7833
> Find your local number: https://linaro-org.zoom.us/u/acSg35gbut
>
> Regards,
> Jammy
> --
> Bdds-dev mailing list
> Bdds-dev(a)op-lists.linaro.org
> https://op-lists.linaro.org/mailman/listinfo/bdds-dev
>
Hi All,
With recent changes in the industry, BigTop is getting more momentum as an
open source distro for Big Data components like Hadoop, Spark and HBase.
We're going to host a meeting to discuss how we can work together in the
community to enhance it and resolve the gaps/issues. For example:
- What components are still missed?
- What component versions should be used?
- Any problems for using with Ambari?
Please check the meeting info below, and welcome to join.
============
Topic: BigTop open discussion
Time: May 28, 2021 10:00 AM CST (UTC+8)
Join Zoom Meeting
https://linaro-org.zoom.us/j/98394347833
Meeting ID: 983 9434 7833
One tap mobile
+16699009128,,98394347833# US (San Jose)
+12532158782,,98394347833# US (Tacoma)
Dial by your location
+1 669 900 9128 US (San Jose)
+1 253 215 8782 US (Tacoma)
+1 301 715 8592 US (Washington DC)
+1 312 626 6799 US (Chicago)
+1 346 248 7799 US (Houston)
+1 646 558 8656 US (New York)
888 788 0099 US Toll-free
877 853 5247 US Toll-free
Meeting ID: 983 9434 7833
Find your local number: https://linaro-org.zoom.us/u/acSg35gbut
Regards,
Jammy
Hi,
It's pretty helpful to support RDMA for hadoop, Spark , Kafka, etc.
But from http://hibd.cse.ohio-state.edu/downloads, it just offers the
tarball for x86 and '*RDMA for Apache Hadoop 3.x Userguide'* is not
available.
IMHO, we should confirm whether HiBD <http://hibd.cse.ohio-state.edu/> is
open-source or not.
Maybe it would be better to set up a talk-channel with guys in
OHIO-State-University first.
Thanks!
BRs,
Yuqi
On Tue, 8 Sep 2020 at 21:49, Yangjian (nt) via Bdds-dev <
bdds-dev(a)op-lists.linaro.org> wrote:
>
> Hi
>
> Yes, It can improve the arm ecology and some big data components
> performance, at some times looking at the views of other partners.
>
>
> 发件人: Ganesh Raju [mailto:ganesh.raju@linaro.org]
> 发送时间: 2020年9月8日 21:39
> 收件人: Yangjian (nt) <yangjian.yx(a)huawei.com>
> 抄送: bdds-dev(a)op-lists.linaro.org
> 主题: Re: [Bdds-dev] RDMA capability for Bigdata components
>
> Hi
> I think it is good to invest some time onto this. We had raised this to
> steering committee a while back (almost 1.5 years ago). We need to relook
> at it.
>
> Thanks,
> Ganesh
>
> On Thu, Sep 3, 2020 at 11:18 AM Yangjian (nt) via Bdds-dev <
> bdds-dev(a)op-lists.linaro.org<mailto:bdds-dev@op-lists.linaro.org>> wrote:
> Hi ALL:
>
> To raise a topic, whether the RDMA capability can be supported on ARM
> in BigTop project for some bigData components, refer to the link as
> fellows, Big Data Components + RDMA can improve the performance, such as
> Spark and Hadoop.
>
> http://hibd.cse.ohio-state.edu/
> --
> Bdds-dev mailing list
> Bdds-dev(a)op-lists.linaro.org<mailto:Bdds-dev@op-lists.linaro.org>
> https://op-lists.linaro.org/mailman/listinfo/bdds-dev
>
>
> --
> IRC: ganeshraju@#linaro on irc.freenode.ne<http://irc.freenode.net/>t
> --
> Bdds-dev mailing list
> Bdds-dev(a)op-lists.linaro.org
> https://op-lists.linaro.org/mailman/listinfo/bdds-dev
>
Hi
I think it is good to invest some time onto this. We had raised this to
steering committee a while back (almost 1.5 years ago). We need to relook
at it.
Thanks,
Ganesh
On Thu, Sep 3, 2020 at 11:18 AM Yangjian (nt) via Bdds-dev <
bdds-dev(a)op-lists.linaro.org> wrote:
> Hi ALL:
>
> To raise a topic, whether the RDMA capability can be supported on ARM
> in BigTop project for some bigData components, refer to the link as
> fellows, Big Data Components + RDMA can improve the performance, such as
> Spark and Hadoop.
>
> http://hibd.cse.ohio-state.edu/
> --
> Bdds-dev mailing list
> Bdds-dev(a)op-lists.linaro.org
> https://op-lists.linaro.org/mailman/listinfo/bdds-dev
>
--
IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Hi ALL:
To raise a topic, whether the RDMA capability can be supported on ARM in BigTop project for some bigData components, refer to the link as fellows, Big Data Components + RDMA can improve the performance, such as Spark and Hadoop.
http://hibd.cse.ohio-state.edu/