powered_by.md

layout	title	description
article	Powered by	List of projects powered by Apache Arrow

Project and Product Names Using "Apache Arrow"

Organizations creating products and projects for use with Apache Arrow, along with associated marketing materials, should take care to respect the trademark in "Apache Arrow" and its logo. Please refer to ASF Trademarks Guidance and associated FAQ for comprehensive and authoritative guidance on proper usage of ASF trademarks.

Names that do not include "Apache Arrow" at all have no potential trademark issue with the Apache Arrow project. This is recommended.

Names like "Apache Arrow BigCoProduct" are not OK, as are names including "Apache Arrow" in general. The above links, however, describe some exceptions, like for names such as "BigCoProduct, powered by Apache Arrow" or "BigCoProduct for Apache Arrow".

It is common practice to create software identifiers (Maven coordinates, module names, etc.) like "arrow-foo". These are permitted. Nominative use of trademarks in descriptions is also always allowed, as in "BigCoProduct is a widget for Apache Arrow".

Projects and documents that want to include a logo for Apache Arrow should use the official logo, and adhere to the guidelines listed on the [Visual Identity]({{ site.baseurl }}/visual_identity/) page:

Projects Powered By Apache Arrow

To add yourself to the list, please open a pull request adding your organization name, URL, a list of which Arrow components you are using, and a short description of your use case.

• Apache Parquet: A columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language. The C++ and Java implementation provide vectorized reads and write to/from Arrow data structures.
• Apache Spark: Apache Spark™ is a fast and general engine for large-scale data processing. Spark uses Apache Arrow to
  1. improve performance of conversion between Spark DataFrame and pandas DataFrame
  2. enable a set of vectorized user-defined functions (pandas_udf) in PySpark.
• ArcPy: ArcPy is Esri’s comprehensive and powerful API for working within the ArcGIS suite of products to perform and automate spatial analysis, data management, and conversion tasks (license required). ArcPy supports Arrow Tables as input and output.
• AWS Data Wrangler: Extends the power of Pandas library to AWS connecting DataFrames and AWS data related services such as Amazon Redshift, AWS Glue, Amazon Athena, Amazon EMR, Amazon QuickSight, etc.
• Bodo: Bodo is a universal Python analytics engine that democratizes High Performance Computing (HPC) architecture for mainstream enterprises, allowing Python analytics workloads to scale efficiently. Bodo uses Arrow to support I/O for Parquet files, as well as internal support for data operations.
• ClickHouse: An open-source analytical database management system. ClickHouse is using Apache Arrow for data import and export, and for direct querying of external datasets in Arrow, ArrowStream, Parquet and ORC formats.
• CloudQuery: An open-source high performance ELT framework powered by Apache Arrow's type system.
• Cylon: An open-source high performance distributed data processing library that can be seamlessly integrated with existing Big Data and AI/ML frameworks. Cylon uses Arrow memory format and exposes language bindings to C++, Java, and Python.
• Dask: Python library for parallel and distributed execution of dynamic task graphs. Dask supports using pyarrow for accessing Parquet files
• Data Preview: Data Preview is a Visual Studio Code extension for viewing text and binary data files. Data Preview uses Arrow JS API for loading, transforming and saving Arrow data files and schemas.
• delta-rs: A native Rust library for Delta Lake, with bindings to Python. It can be integrated with Apache Arrow, increasing the efficiency of data exchange over the network
• Dremio: A self-service data platform. Dremio makes it easy for users to discover, curate, accelerate, and share data from any source. It includes a distributed SQL execution engine based on Apache Arrow. Dremio reads data from any source (RDBMS, HDFS, S3, NoSQL) into Arrow buffers, and provides fast SQL access via ODBC, JDBC, and REST for BI, Python, R, and more (all backed by Apache Arrow).
• Falcon: An interactive data exploration tool with coordinated views. Falcon loads Arrow files using the Arrow JavaScript module. Since Arrow does not need to be parsed (like text-based formats like CSV and JSON), startup cost is significantly minimized.
• FASTDATA.io: Plasma Engine (unrelated to Arrow's Plasma In-Memory Object Store) exploits the massive parallel processing power of GPUs for stream and batch processing. It supports Arrow as input and output, uses Arrow internally to maximize performance, and can be used with existing Apache Spark™ APIs.
• Fletcher: Fletcher is a framework that can integrate FPGA accelerators with tools and frameworks that use the Apache Arrow in-memory format. From a set of Arrow Schemas, Fletcher generates highly optimized hardware structures that allow accelerator kernels to read and write RecordBatches at system bandwidth through easy-to-use interfaces.
• GeoMesa: A suite of tools that enables large-scale geospatial query and analytics on distributed computing systems. GeoMesa supports query results in the Arrow IPC format, which can then be used for in-browser visualizations and/or further analytics.
• GOAI: Open GPU-Accelerated Analytics Initiative for Arrow-powered analytics across GPU tools and vendors
• graphique GraphQL service for arrow tables and parquet data sets. The schema for a query API is derived automatically.
• Graphistry: Supercharged Visual Investigation Platform used by teams for security, anti-fraud, and related investigations. The Graphistry team uses Arrow in its NodeJS GPU backend and client libraries, and is an early contributing member to GOAI and Arrow[JS] focused on bringing these technologies to the enterprise.
• GreptimeDB: GreptimeDB is an open-source time-series database with a special focus on scalability, analytical capabilities and efficiency. It's designed to work on infrastructure of the cloud era, and users benefit from its elasticity and commodity storage. GreptimeDB uses Apache Arrow as the memory model and Apache Parquet as the persistent file format.
• HASH: HASH is an open-core platform for building, running, and learning from simulations, with an in-browser IDE. HASH Engine uses Apache Arrow to power the datastore for simulation state during computation, enabling zero-copy data
• Hugging Face Datasets: A machine learning datasets library and hub for accessing, processing and sharing datasets for audio, computer vision, natural language processing, and tabular tasks. Dataset objects are wrappers around Arrow Tables and memory-mapped from disk to support out-of-core parallel processing for machine learning workflows.
• iceburst: A real-time data lake for monitoring and security built directly on top of Amazon S3. Our approach is simple: ingest the OpenTelemetry data in an S3 bucket as Parquet files in Iceberg table format and query them using DuckDB with milliseond retrieval and zero egress cost. Parquet is converted to Arrow format in-memory enhancing both speed and efficiency.
• InAccel: A machine learning acceleration framework which leverages FPGAs-as-a-service. InAccel supports dataframes backed by Apache Arrow to serve as input for our implemented ML algorithms. Those dataframes can be accessed from the FPGAs with a single DMA operation by implementing a shared memory communication schema.
• InfluxDB IOx: InfluxDB IOx is an open source time series database written in Rust. It is the future core of InfluxDB; supporting industry standard SQL, InfluxQL, and Flux. IOx uses Apache Arrow as its in-memory format, Apache Parquet as its persistence format and Apache Arrow Flight for RPC.
• Kaskada: An open source event processing engine written in Rust and built on Apache Arrow.
• libgdf: A C library of CUDA-based analytics functions and GPU IPC support for structured data. Uses the Arrow IPC format and targets the Arrow memory layout in its analytic functions. This work is part of the GPU Open Analytics Initiative
• MATLAB: A numerical computing environment for engineers and scientists. MATLAB uses Apache Arrow to support reading and writing Parquet and Feather files.
• OmniSci (formerly MapD): In-memory columnar SQL engine designed to run on both GPUs and CPUs. OmniSci supports Arrow for data ingest and data interchange via CUDA IPC handles. This work is part of the GPU Open Analytics Initiative
• OpenObserve: Petabyte scale observability tool for logs, metrics, and traces with visualizations. High focus on usability and simplicity. Supports opentelemetry and many existing log and metrics forwarders.
• pandas: data analysis toolkit for Python programmers. pandas supports reading and writing Parquet files using pyarrow. Several pandas core developers are also contributors to Apache Arrow.
• pantab: Allows high performance read/writes of popular dataframe libraries like pandas, polars, pyarrow, etc... to/from Tableau's Hyper database. pantab uses nanoarrow and the Arrow PyCapsule interface to make that exchange process seamless.
• Parseable: Log analytics platform built for scale and usability. Ingest logs from anywhere and unify logs with Parseable. Parseable uses Arrow as the intermediary, in-memory data format for log data ingestion.
• Perspective: Perspective is a streaming data visualization engine in JavaScript for building real-time & user-configurable analytics entirely in the browser.
• Petastorm: Petastorm enables single machine or distributed training and evaluation of deep learning models directly from datasets in Apache Parquet format. Petastorm supports popular Python-based machine learning (ML) frameworks such as Tensorflow, Pytorch, and PySpark. It can also be used from pure Python code.
• Polars: Polars is a blazingly fast DataFrame library and query engine that aims to utilize modern hardware efficiently. (e.g. multi-threading, SIMD vectorization, hiding memory latencies). Polars is built upon Apache Arrow and uses its columnar memory, compute kernels, and several IO utilities. Polars is written in Rust and available in Rust and Python.
• protarrow: A Python library for converting from Apache Arrow to Protocol Buffers and back.
• Quilt Data: Quilt is a data package manager, designed to make managing data as easy as managing code. It supports Parquet format via pyarrow for data access.
• Ray: A flexible, high-performance distributed execution framework with a focus on machine learning and AI applications. Uses Arrow to efficiently store Python data structures containing large arrays of numerical data. Data can be accessed with zero-copy by multiple processes using the Plasma shared memory object store which originated from Ray and is part of Arrow now.
• Red Data Tools: A project that provides data processing tools for Ruby. It provides Red Arrow that is a Ruby bindings of Apache Arrow based on Apache Arrow GLib. Red Arrow is a core library for it. It also provides many Ruby libraries to integrate existing Ruby libraries with Apache Arrow. They use Red Arrow.
• SciDB: Paradigm4's SciDB is a scalable, scientific database management system that helps researchers integrate and analyze diverse, multi-dimensional, high resolution data - like genomic, clinical, images, sensor, environmental, and IoT data - all in one analytical platform. SciDB streaming and accelerated_io_tools are powered by Apache Arrow.
• Spice.ai OSS: A unified SQL query interface and portable runtime built in Rust to locally materialize, accelerate, and query datasets from any database, data warehouse, or data lake. Spice.ai OSS uses Arrow along with DataFusion internally, and supports Flight and Flight SQL connectivity.
• Squey: Squey is an open-source visualization software designed to interactively explore and understand large amounts of columnar data. It uses Apache Arrow C++ library (with Arrow Compute) to import and export Parquet files.
• TileDB: TileDB is an open-source, cloud-optimized engine for storing and accessing dense/sparse multi-dimensional arrays and dataframes. It is an embeddable C++ library that works on Linux, macOS, and Windows, which comes with numerous APIs and integrations. We use Arrow in our TileDB-VCF project for genomics to achieve zero-copying when accessing TileDB data from Spark and Dask.
• Turbodbc: Python module to access relational databases via the Open Database Connectivity (ODBC) interface. It provides the ability to return Arrow Tables and RecordBatches in addition to the Python Database API Specification 2.0.
• UKV: Open NoSQL binary database interface, with support for LevelDB, RocksDB, UDisk, and in-memory Key-Value Stores. It extends their functionality to support Document Collections, Graphs, and Vector Search, similar to RedisJSON, RedisGraph, and RediSearch, and brings familiar structured bindings on top, mimicking tools like pandas and NetworkX. All UKV interfaces are compatible with Apache Arrow columnar format, which minimizes copies when passing data between different language runtimes. UKV also uses Apache Arrow Flight RPC for client-server communication.
• Vaex: Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualize and explore big tabular data at a billion rows per second.
• VAST: A network telemetry engine for data-driven security investigations. VAST uses Arrow as standardized data plane to provide a high-bandwidth output path for downstream analytics. This makes it easy and efficient to access security data via pyarrow and other available bindings.