DESCRIPTION

Package: ff
Version: 4.5.0
Date: 2024-09-17
Title: Memory-Efficient Storage of Large Data on Disk and Fast Access Functions
Authors@R: c(person(given = "Daniel",
                      family = "Adler",
                      role = "aut"),
               person(given = "Christian",
                      family = "Gläser",
                      role = "ctb"),
               person(given = "Oleg",
                      family = "Nenadic",
                      role = "ctb"),
               person(given = "Jens",
                      family = "Oehlschlägel",
                      role = c("aut", "cre"),
                      email = "Jens.Oehlschlaegel@truecluster.com"),
               person(given = "Martijn",
                      family = "Schuemie",
                      role = "ctb"),
               person(given = "Walter",
                      family = "Zucchini",
                      role = "ctb"))
Author: Daniel Adler [aut],
	Christian Gläser [ctb],
	Oleg Nenadic  [ctb], 
	Jens Oehlschlägel [aut, cre],
	Martijn Schuemie [ctb],
	Walter Zucchini  [ctb]
Maintainer: Jens Oehlschlägel <Jens.Oehlschlaegel@truecluster.com>
Depends: R (>= 3.4.0), bit (>= 4.0.0), utils
Suggests: biglm, testthat (>= 0.11.0), markdown
Description: The ff package provides data structures that are stored on
	disk but behave (almost) as if they were in RAM by transparently 
	mapping only a section (pagesize) in main memory - the effective 
	virtual memory consumption per ff object. ff supports R's standard 
	atomic data types 'double', 'logical', 'raw' and 'integer' and 
	non-standard atomic types boolean (1 bit), quad (2 bit unsigned), 
	nibble (4 bit unsigned), byte (1 byte signed with NAs), ubyte (1 byte 
	unsigned), short (2 byte signed with NAs), ushort (2 byte unsigned), 
	single (4 byte float with NAs). For example 'quad' allows efficient 
	storage of genomic data as an 'A','T','G','C' factor. The unsigned 
	types support 'circular' arithmetic. There is also support for 
	close-to-atomic types 'factor', 'ordered', 'POSIXct', 'Date' and 
	custom close-to-atomic types. 
	ff not only has native C-support for vectors, matrices and arrays 
	with flexible dimorder (major column-order, major row-order and 
	generalizations for arrays). There is also a ffdf class not unlike 
	data.frames and import/export filters for csv files.
	ff objects store raw data in binary flat files in native encoding,
	and complement this with metadata stored in R as physical and virtual
	attributes. ff objects have well-defined hybrid copying semantics, 
	which gives rise to certain performance improvements through 
	virtualization. ff objects can be stored and reopened across R 
	sessions. ff files can be shared by multiple ff R objects 
	(using different data en/de-coding schemes) in the same process 
	or from multiple R processes to exploit parallelism. A wide choice of 
	finalizer options allows to work with 'permanent' files as well as 
	creating/removing 'temporary' ff files completely transparent to the 
	user. On certain OS/Filesystem combinations, creating the ff files
	works without notable delay thanks to using sparse file allocation.
	Several access optimization techniques such as Hybrid Index 
	Preprocessing and Virtualization are implemented to achieve good 
	performance even with large datasets, for example virtual matrix 
	transpose without touching a single byte on disk. Further, to reduce 
	disk I/O, 'logicals' and non-standard data types get stored native and 
	compact on binary flat files i.e. logicals take up exactly 2 bits to 
	represent TRUE, FALSE and NA. 
	Beyond basic access functions, the ff package also provides 
	compatibility functions that facilitate writing code for ff and ram 
	objects and support for batch processing on ff objects (e.g. as.ram, 
	as.ff, ffapply). ff interfaces closely with functionality from package 
	'bit': chunked looping, fast bit operations and coercions between 
	different objects that can store subscript information ('bit', 
	'bitwhich', ff 'boolean', ri range index, hi hybrid index). This allows
	to work interactively with selections of large datasets and quickly 
	modify selection criteria. 
	Further high-performance enhancements can be made available upon request. 
License: GPL-2 | GPL-3 | file LICENSE
LazyLoad: yes
ByteCompile: yes
Encoding: UTF-8
URL: https://github.com/truecluster/ff