- Bdds-dev - op-lists.linaro.org

答复: Some problems with bigtop are adapted to openEuler system, pls help to reply, tks.

by Yangjian (nt)

Hi, all According to the discussion, the PR generated by the openEuler docker image has been provided. It support the build of puppet and slaves docker, the base docker can pull from the openEuler web. Add OpenEuler 22.03 support PR address: https://github.com/apache/bigtop/pull/1051 Issue address: https://issues.apache.org/jira/browse/BIGTOP-3873 On this, we can upload the patch files of components one by one, for support openEuler OS. Please check it out and give suggestions, thx. 发件人: Yangjian (nt) 发送时间: 2022年9月26日 19:19 收件人: bdds-dev(a)op-lists.linaro.org 抄送: chenqiang (N) <mac.chenqiang(a)huawei.com>; Guodong Xu <guodong.xu(a)linaro.org>; Luoliyuan <luoliyuan(a)huawei.com> 主题: [Bdds-dev] Some problems with bigtop are adapted to openEuler system, pls help to reply, tks. English： 1. Where can I get the component installation and deployment information during the smoke-test? Such as include component startup options and commands? What options do the test components need to configure? 2. Is the smoke-test using a single machine or a cluster? 3. Smoke-test test cases without Ranger 2.2.0 components. 4. The gpdb 5.28.5 component does not support python3 when compiled. 5. ycsb 0.17.0 component smoke test error log: File "/usr/lib/ycsb/bin/ycsb", line 228 except subprocess.CalledProcessError, err: ^ SyntaxError: invalid syntax 6．We solved the problem and raised a PR: https://github.com/apache/bigtop/pull/1025 Chinese: 1、冒烟测试过程中组件安装部署信息在哪儿可以获取到？比如包括组件启动选项和命令是什么？测试组件需要配置哪些选项？ 2、冒烟测试是用单机还是集群？ 3、冒烟测试没有ranger组件的测试用例 4、gpdb 5.28.5 组件编译时不支持python3 5、ycsb 0.17.0 组件冒烟测试报错： File "/usr/lib/ycsb/bin/ycsb", line 228 except subprocess.CalledProcessError, err: ^ SyntaxError: invalid syntax 6、我们解决问题提了一个PR： https://github.com/apache/bigtop/pull/1025 Best Regards

1 year, 10 months

1
0
0 0

Some problems with bigtop are adapted to openEuler system, pls help to reply, tks.

by Yangjian (nt)

English： 1. Where can I get the component installation and deployment information during the smoke-test? Such as include component startup options and commands? What options do the test components need to configure? 2. Is the smoke-test using a single machine or a cluster? 3. Smoke-test test cases without Ranger 2.2.0 components. 4. The gpdb 5.28.5 component does not support python3 when compiled. 5. ycsb 0.17.0 component smoke test error log: File "/usr/lib/ycsb/bin/ycsb", line 228 except subprocess.CalledProcessError, err: ^ SyntaxError: invalid syntax 6．We solved the problem and raised a PR: https://github.com/apache/bigtop/pull/1025 Chinese: 1、冒烟测试过程中组件安装部署信息在哪儿可以获取到？比如包括组件启动选项和命令是什么？测试组件需要配置哪些选项？ 2、冒烟测试是用单机还是集群？ 3、冒烟测试没有ranger组件的测试用例 4、gpdb 5.28.5 组件编译时不支持python3 5、ycsb 0.17.0 组件冒烟测试报错： File "/usr/lib/ycsb/bin/ycsb", line 228 except subprocess.CalledProcessError, err: ^ SyntaxError: invalid syntax 6、我们解决问题提了一个PR： https://github.com/apache/bigtop/pull/1025 Best Regards

1 year, 10 months

2
2
0 0

Bigtop OpenEuler support

by Yangjian (nt)

Hi ALL： 1. Which version of Bigtop should be based on to support openEuler OS ？(3.1.1 or 3.2.2 ?) Is there a development plan and roadmap for this release? 2. The current stable version of openEuler OS is 22.03, and some system software version information is as follows. What should I do if it is inconsistent with your planned version? For example, the higher python version is 3.9.9, and hadoop compilation requires python 2.7 version. [cid:image001.png@01D8B661.AA6A3640]

2 years

1
0
0 0

Fwd: [ANNOUNCE] Apache Bigtop 3.0.0 released

by Ganesh Raju

---------- Forwarded message --------- From: Kengo Seki <sekikn(a)apache.org> Date: Mon, Oct 25, 2021 at 11:32 AM Subject: [ANNOUNCE] Apache Bigtop 3.0.0 released To: <announce(a)apache.org>, <user(a)bigtop.apache.org>, dev < dev(a)bigtop.apache.org> On behalf of the Apache Bigtop team, I'd love to announce the general availability of the Bigtop 3.0.0 release. The release is available here: https://bigtop.apache.org/download.html#releases A few highlights of this release include: * Major components such as Hadoop, Hive and Spark are finally upgraded to 3.x. * Other components are also upgraded so that they are compatible with the above ones and users can enjoy their enhancement. * Fedora 33 and Ubuntu 20.04 are newly added as the supported distros. * In addition to x86_64 and aarch64, ppc64le is also supported, which was not in Bigtop 1.5.0. With Bigtop 3.0.0 the community continues to deliver the most advanced big data stack to date. More details about 3.0.0 release are here: https://bigtop.apache.org/release-notes.html Deploying Bigtop is easy: grab the repo/list file for your favorite Linux distribution: https://www.apache.org/dyn/closer.lua/bigtop/bigtop-3.0.0/repos/ and you'll be running your very own bigdata cluster in no time! We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at: https://bigtop.apache.org Lastly, I want to emphasize that this is a collaborative work done by project contributors and other communities, who continue to devote time to make Bigtop a better software. Thank you all for making this release possible! Regards, Kengo Seki, Bigtop 3.0.0 Release Manager -- IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t

2 years, 11 months

1
0
0 0

Fwd: The Apache Drill Project Announces Apache(R) DrillTM v1.19 Milestone Release

by Ganesh Raju

Announcement of Apache Drill now having official ARM release ---------- Forwarded message --------- From: Sally Khudairi <sk(a)apache.org> Date: Mon, Aug 30, 2021 at 8:10 AM Subject: The Apache Drill Project Announces Apache® DrillTM v1.19 Milestone Release To: Apache Announce List <announce(a)apache.org> [this announcement is available online at https://s.apache.org/bfhy6 ] Open Source, enterprise-grade, schema-free Big Data SQL query engine used by thousands of organizations, including Ant Group, Cisco, Ericsson, Intuit, MicroStrategy, Tableau, TIBCO, TransUnion, Twitter, and more. Wilmington, DE —30 August 2021— The Apache Drill Project announced the release of Apache® DrillTM v1.19, the schema-free Big Data SQL query engine for Apache Hadoop®, NoSQL, and Cloud storage. "Drill 1.19 is our biggest release ever," said Charles Givre, Vice President of Apache Drill. "With an already short learning curve, Drill 1.19 makes it even easier for users to quickly query, analyze, and visualize data from disparate sources and complex data sets.” An "SQL-on-Hadoop" engine, Apache Drill is easy to deploy, highly performant, able to quickly process trillions of records, and scalable from a single laptop to a 1000-node cluster. With its schema-free JSON model (the first distributed SQL query engine of its kind), Drill is able to query complex semi-structured data in situ without requiring users to define schemas or transform data. It provides plug-and-play integration with existing Hive and HBase deployments, and is extensible out-of-the-box to access multiple data sources, such as S3 and Apache HDFS, HBase, and Hive. Additionally, Drill can directly query data from REST APIs to include platforms like SalesForce and ServiceNow. Drill supports the ANSI SQL 2003 standard syntax ecosystem as well as dozens of NoSQL databases and file systems, including Apache HBase, MongoDB, Elasticsearch, Cassandra, REST APIs, , HDFS, MapR-FS, Amazon S3, Azure Blob Storage, Google Cloud Storage, NAS, local files, and more. Drill leverages familiar BI tools (such as Apache Superset, Tableau, MicroStrategy, QlikView and Excel) as well as data virtualization and visualization tools, and runs interactive queries on Hive tables with different Hive metastores. Apache Drill v1.19 Drill is designed from the ground up to support high-performance analysis on rapidly evolving data on modern Big Data applications. v1.19 reflects more than 100 changes, improvements, and new features that include: - New Connectors for Apache Cassandra, Elasticsearch, and Splunk - New Format Reader for XML without schemas - Added Avro support for Kafka plugin - Integrated password vault for secure credential storage - Support for Linux ARM64 systems - Added limit pushdowns for file systems, HTTP REST APIs and MongoDB - Added streaming for Drill's REST API - Integration with Apache Airflow Developers, analysts, business users, and data scientists use Apache Drill for data exploration and analysis for its enterprise-grade reliability, security, and performance. Drill's flexibility and ease-of-use have attracted thousands of users that include Ant Group, Cardlytics, Cisco, Ericsson, Intuit, MicroStrategy, Qlik, Tableau, TIBCO, TransUnion, Twitter, National University of Singapore, and more. "Individuals, businesses, and organizations of all types rely on Apache Drill's rich functionality," added Givre. "We invite everyone to participate in our user and developer lists as well as our Slack channel, and contribute to the project to build on our momentum and help improve the future experience for all Drill users." Catch Apache Drill in action at ApacheCon@Home, taking place online 21-23 September 2021. For more information and to register, visit https://www.apachecon.com/ . Availability and Oversight Apache Drill software is released under the Apache License v2.0 and is overseen by a volunteer, self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. About Apache Drill Apache Drill is the Open Source, schema-free Big Data SQL query engine for Apache Hadoop, NoSQL, and Cloud storage. For more information, including documentation and ways to become involved with Apache Drill, visit http://drill.apache.org/ , https://twitter.com/ApacheDrill , and https://apache-drill.slack.com/ . © The Apache Software Foundation. "Apache", "Drill", "Apache Drill", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners. # # # NOTE: you are receiving this message because you are subscribed to the announce(a)apache.org distribution list. To unsubscribe, send email from the recipient account to announce-unsubscribe(a)apache.org with the word "Unsubscribe" in the subject line. -- IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t

3 years

1
0
0 0

Re: [Bdds-dev] BigTop open discussion

by Jammy Zhou

Hi All, Thanks for joining the call today. Below are some notes from this meeting. *Attendees*: Gugang, Shenshu, Xiaoqi Ma, Jian Yang, Yuqi, Jun He, Ganesh, Jammy, Elsie *Notes*: - Current BigTop release is v1.5, now moving to 3.0 to integrate Hadoop v3.x, and Ambari mpack support is in progress - ApacheCon 2021 is coming (Sep. 21~23), and some announcement is planned about BigTop release v3.0 - People who use HDP and CDH start looking at BigTop - Future plans and ideas for BigTop (need to prioritize) - Kubernetes support - More components to be added like Apache Beam, Arrow, Ranger, etc - Looking forward to feedback from Huawei and other vendors about BigTop and other Big Data and Data Science projects - JIRA link for BigTop v3.0 release https://issues.apache.org/jira/browse/BIGTOP-3471 - BigTop mailing list http://bigtop.apache.org/mail-lists.html - Shenshu shared some requirements (by comparing to Ambari+HDP) - deploy all components by Ambari - support openEuler/Kylin/UOS operating systems - upgrade some components to new versions such as Hadoop/Hive/Hbase/etc - add Pig/Storm/Ranger/Atlas/Knox/Accumulo/Druid/Impala/Kudu - provide a install package (tar.gz) that contains all the rpms/debs - every component should release a tar.gz package - Most components will be upgraded for BigTop v3.0, and the missing components can be interesting to add - Solr will be upgraded to 8.7 for BigTop v3.0 - Gang: current CI has only x86 and CentOS for components like Hadoop, Spark, Hbase in the BigTop community, doesn’t have Aarch64 - Jun: the work for v3.0 is still ongoing, some CI support is disabled temporarily, for the final release, x86/Aarch64/PowerPC will all be supported - Gang: Ambari mpack can only detect several components now, which is quite limited - Yuqi: Ambari will be upgraded to 2.7.5, there will be some work for mpack, we can do together if there is some special requirements (the core components have high priority) - Hadoop 3.3 has official support for AArch64 - Impala and Kudu will be required (please raise a ticket in the JIRA link for BIGTOP-3471), similar for other requirements - For BigTop, it is hard to have a single tar file for all components. Encourage to participate in the BigTop community (e.g, mailing list) for related discussions - The BigTop community is open to add more distro support (like openEuler). Is there some docker image for openEuler? - Yes, docker image can be found in the docker_img folder of openEuler repo, like https://repo.openeuler.org/openEuler-20.03-LTS-SP1/docker_img/ Regards, Jammy On Wed, 26 May 2021 at 20:43, Jammy Zhou via Bdds-dev < bdds-dev(a)op-lists.linaro.org> wrote: > Hi All, > > With recent changes in the industry, BigTop is getting more momentum as an > open source distro for Big Data components like Hadoop, Spark and HBase. > > We're going to host a meeting to discuss how we can work together in the > community to enhance it and resolve the gaps/issues. For example: > > - What components are still missed? > - What component versions should be used? > - Any problems for using with Ambari? > > Please check the meeting info below, and welcome to join. > ============ > Topic: BigTop open discussion > Time: May 28, 2021 10:00 AM CST (UTC+8) > > Join Zoom Meeting > https://linaro-org.zoom.us/j/98394347833 > > Meeting ID: 983 9434 7833 > One tap mobile > +16699009128,,98394347833# US (San Jose) > +12532158782,,98394347833# US (Tacoma) > > Dial by your location > +1 669 900 9128 US (San Jose) > +1 253 215 8782 US (Tacoma) > +1 301 715 8592 US (Washington DC) > +1 312 626 6799 US (Chicago) > +1 346 248 7799 US (Houston) > +1 646 558 8656 US (New York) > 888 788 0099 US Toll-free > 877 853 5247 US Toll-free > Meeting ID: 983 9434 7833 > Find your local number: https://linaro-org.zoom.us/u/acSg35gbut > > Regards, > Jammy > -- > Bdds-dev mailing list > Bdds-dev(a)op-lists.linaro.org > https://op-lists.linaro.org/mailman/listinfo/bdds-dev >

3 years, 3 months

1
0
0 0

BigTop open discussion

by Jammy Zhou

Hi All, With recent changes in the industry, BigTop is getting more momentum as an open source distro for Big Data components like Hadoop, Spark and HBase. We're going to host a meeting to discuss how we can work together in the community to enhance it and resolve the gaps/issues. For example: - What components are still missed? - What component versions should be used? - Any problems for using with Ambari? Please check the meeting info below, and welcome to join. ============ Topic: BigTop open discussion Time: May 28, 2021 10:00 AM CST (UTC+8) Join Zoom Meeting https://linaro-org.zoom.us/j/98394347833 Meeting ID: 983 9434 7833 One tap mobile +16699009128,,98394347833# US (San Jose) +12532158782,,98394347833# US (Tacoma) Dial by your location +1 669 900 9128 US (San Jose) +1 253 215 8782 US (Tacoma) +1 301 715 8592 US (Washington DC) +1 312 626 6799 US (Chicago) +1 346 248 7799 US (Houston) +1 646 558 8656 US (New York) 888 788 0099 US Toll-free 877 853 5247 US Toll-free Meeting ID: 983 9434 7833 Find your local number: https://linaro-org.zoom.us/u/acSg35gbut Regards, Jammy

3 years, 3 months

1
0
0 0

Re: [Bdds-dev] 答复: RDMA capability for Bigdata components

by Yuqi Gu

Hi, It's pretty helpful to support RDMA for hadoop, Spark , Kafka, etc. But from http://hibd.cse.ohio-state.edu/downloads, it just offers the tarball for x86 and '*RDMA for Apache Hadoop 3.x Userguide'* is not available. IMHO, we should confirm whether HiBD <http://hibd.cse.ohio-state.edu/> is open-source or not. Maybe it would be better to set up a talk-channel with guys in OHIO-State-University first. Thanks! BRs, Yuqi On Tue, 8 Sep 2020 at 21:49, Yangjian (nt) via Bdds-dev < bdds-dev(a)op-lists.linaro.org> wrote: > > Hi > > Yes, It can improve the arm ecology and some big data components > performance, at some times looking at the views of other partners. > > > 发件人: Ganesh Raju [mailto:ganesh.raju@linaro.org] > 发送时间: 2020年9月8日 21:39 > 收件人: Yangjian (nt) <yangjian.yx(a)huawei.com> > 抄送: bdds-dev(a)op-lists.linaro.org > 主题: Re: [Bdds-dev] RDMA capability for Bigdata components > > Hi > I think it is good to invest some time onto this. We had raised this to > steering committee a while back (almost 1.5 years ago). We need to relook > at it. > > Thanks, > Ganesh > > On Thu, Sep 3, 2020 at 11:18 AM Yangjian (nt) via Bdds-dev < > bdds-dev(a)op-lists.linaro.org<mailto:bdds-dev@op-lists.linaro.org>> wrote: > Hi ALL： > > To raise a topic, whether the RDMA capability can be supported on ARM > in BigTop project for some bigData components, refer to the link as > fellows, Big Data Components + RDMA can improve the performance, such as > Spark and Hadoop. > > http://hibd.cse.ohio-state.edu/ > -- > Bdds-dev mailing list > Bdds-dev(a)op-lists.linaro.org<mailto:Bdds-dev@op-lists.linaro.org> > https://op-lists.linaro.org/mailman/listinfo/bdds-dev > > > -- > IRC: ganeshraju@#linaro on irc.freenode.ne<http://irc.freenode.net/>t > -- > Bdds-dev mailing list > Bdds-dev(a)op-lists.linaro.org > https://op-lists.linaro.org/mailman/listinfo/bdds-dev >

4 years

2
1
0 0

Re: [Bdds-dev] RDMA capability for Bigdata components

by Ganesh Raju

Hi I think it is good to invest some time onto this. We had raised this to steering committee a while back (almost 1.5 years ago). We need to relook at it. Thanks, Ganesh On Thu, Sep 3, 2020 at 11:18 AM Yangjian (nt) via Bdds-dev < bdds-dev(a)op-lists.linaro.org> wrote: > Hi ALL： > > To raise a topic, whether the RDMA capability can be supported on ARM > in BigTop project for some bigData components, refer to the link as > fellows, Big Data Components + RDMA can improve the performance, such as > Spark and Hadoop. > > http://hibd.cse.ohio-state.edu/ > -- > Bdds-dev mailing list > Bdds-dev(a)op-lists.linaro.org > https://op-lists.linaro.org/mailman/listinfo/bdds-dev > -- IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t

4 years

2
1
0 0

RDMA capability for Bigdata components

by Yangjian (nt)

Hi ALL： To raise a topic, whether the RDMA capability can be supported on ARM in BigTop project for some bigData components, refer to the link as fellows, Big Data Components + RDMA can improve the performance, such as Spark and Hadoop. http://hibd.cse.ohio-state.edu/

4 years

1
0
0 0

2024

2023

2022

2021

2020

Bdds-dev