add dataset and change names

add TOC
1a1a11a · Jan 1, 2024 · c866b9b · c866b9b
1 parent 26dffad
commit c866b9b
Show file tree

Hide file tree

Showing 5 changed files with 79 additions and 5 deletions.
diff --git a/README.md b/README.md
@@ -3,18 +3,41 @@
 [![build](https://github.com/1a1a11a/libCacheSim/actions/workflows/build.yml/badge.svg)](https://github.com/1a1a11a/libCacheSim/actions/workflows/build.yml)
 
 
+<!-- TOC start (generated with https://github.com/derlin/bitdowntoc) -->
+* [What is libCacheSim](#what-is-libcachesim)
+* [libCacheSim features ](#libcachesim-features)
+* [Supported algorithms](#supported-algorithms)
+* [Build and Install libCacheSim](#build-and-install-libcachesim)
+  + [One-line install](#one-line-install)
+  + [Install dependency](#install-dependency)
+  + [Build libCacheSim](#build-libcachesim)
+* [Usage](#usage)
+  + [cachesim (a high-performance cache simulator)](#cachesim-a-high-performance-cache-simulator)
+  + [Trace analysis](#trace-analysis)
+  + [Using libCacheSim as a library ](#using-libcachesim-as-a-library)
+  + [Extending libCacheSim (new algorithms and trace types)](#extending-libcachesim-new-algorithms-and-trace-types)
+* [Open source cache traces](#open-source-cache-traces)
+* [Questions? ](#questions)
+* [Reference](#reference)
+* [License](#license)
+<!-- TOC end -->
+
+
+<!-- TOC --><a name="news"></a>
 ## News
 * **2023 June**: **QDLP** is available now, see [our paper](https://dl.acm.org/doi/10.1145/3593856.3595887) for details.
 * **2023 Oct**: **[S3-FIFO](https://dl.acm.org/doi/10.1145/3600006.3613147)** and **SIEVE** are available! These are very simple algorithms that are very effective in reducing cache misses. Try them out in libCacheSim and your production!
 ---
 
+<!-- TOC --><a name="what-is-libcachesim"></a>
 ## What is libCacheSim
 * a high-performance **cache simulator** for running cache simulations. 
 * a high-performance and versatile trace analyzer for **analyzing different cache traces**.
 * a high-performance **library** for building cache simulators. 
 
 ---
 
+<!-- TOC --><a name="libcachesim-features"></a>
 ## libCacheSim features 
 * **High performance** - over 20M requests/sec for a realistic trace replay. 
 * **High memory efficiency** - predictable and small memory footprint. 
@@ -25,6 +48,7 @@
 * **Extensible** - easy to support new trace types or eviction algorithms; see [here](/doc/advanced_lib_extend.md).
 ---
 
+<!-- TOC --><a name="supported-algorithms"></a>
 ## Supported algorithms
 cachesim supports the following algorithms:
 * [FIFO](/libCacheSim/cache/eviction/FIFO.c), [LRU](/libCacheSim/cache/eviction/LRU.c), [Clock](/libCacheSim/cache/eviction/Clock.c), [SLRU](/libCacheSim/cache/eviction/SLRU.c)
@@ -45,7 +69,9 @@ cachesim supports the following algorithms:
 ---
 
 
+<!-- TOC --><a name="build-and-install-libcachesim"></a>
 ## Build and Install libCacheSim
+<!-- TOC --><a name="one-line-install"></a>
 ### One-line install
 We provide some scripts for quick installation of libCacheSim. 
 ```bash
@@ -55,11 +81,13 @@ If this does not work, please
 1. let us know what system you are using and what error you get
 2. read the following sections for self-installation.
 
+<!-- TOC --><a name="install-dependency"></a>
 ### Install dependency
 libCacheSim uses [cmake](https://cmake.org/) build system and has a few dependencies: [glib](https://developer.gnome.org/glib/), [tcmalloc](https://github.com/google/tcmalloc), [zstd](https://github.com/facebook/zstd).
 Please see [install.md](/doc/install.md) for instructions on how to install the dependencies. 
 
 
+<!-- TOC --><a name="build-libcachesim"></a>
 ### Build libCacheSim
 cmake recommends **out-of-source build**, so we do it in a new directory:
 ```
@@ -72,16 +100,20 @@ popd;
 ```
 ---
 
+<!-- TOC --><a name="usage"></a>
 ## Usage
+<!-- TOC --><a name="cachesim-a-high-performance-cache-simulator"></a>
 ### cachesim (a high-performance cache simulator)
 After building and installing libCacheSim, `cachesim` should be in the `_build/bin/` directory. 
+<!-- TOC --><a name="basic-usage"></a>
 #### basic usage
 ```
 ./bin/cachesim trace_path trace_type eviction_algo cache_size [OPTION...]
 ```
 
 use `./bin/cachesim --help` to get more information.
 
+<!-- TOC --><a name="run-a-single-cache-simulation"></a>
 #### Run a single cache simulation
 Run the example traces with LRU eviction algorithm and 1GB cache size. 
 
@@ -90,6 +122,7 @@ Run the example traces with LRU eviction algorithm and 1GB cache size.
 ./bin/cachesim ../data/trace.vscsi vscsi lru 1gb 
 ```
 
+<!-- TOC --><a name="run-multiple-cache-simulations-with-different-cache-sizes"></a>
 #### Run multiple cache simulations with different cache sizes
 ```bash
 # Note that there is no space between the cache sizes
@@ -111,6 +144,7 @@ Run the example traces with LRU eviction algorithm and 1GB cache size.
 See [quick start cachesim](/doc/quickstart_cachesim.md) for more usages. 
 
 
+<!-- TOC --><a name="plot-miss-ratio-curve"></a>
 #### Plot miss ratio curve
 You can plot miss ratios of different algorithms and sizes, and plot the miss ratios over time.
 
@@ -125,6 +159,7 @@ python3 plot_mrc_time.py --tracepath ../data/twitter_cluster52.csv --trace-forma
 
 ---
 
+<!-- TOC --><a name="trace-analysis"></a>
 ### Trace analysis
 libCacheSim also has a trace analyzer that provides a lot of useful information about the trace. 
 And it is very fast, designed to work with billions of requests. 
@@ -136,6 +171,7 @@ See [trace analysis](/doc/quickstart_traceAnalyzer.md) for more details.
 
 ---
 
+<!-- TOC --><a name="using-libcachesim-as-a-library"></a>
 ### Using libCacheSim as a library 
 libCacheSim can be used as a library for building cache simulators. 
 For example, you can build a cache cluster with consistent hashing or a multi-layer cache simulator.
@@ -183,6 +219,7 @@ See [here](/doc/advanced_lib.md) for more details, and see [example folder](/exa
 ---
 
 
+<!-- TOC --><a name="extending-libcachesim-new-algorithms-and-trace-types"></a>
 ### Extending libCacheSim (new algorithms and trace types)
 libCacheSim supports *txt*, *csv*, and *binary* traces. We prefer binary traces because it allows libCacheSim to run faster, and the traces are more compact. 
 
@@ -192,19 +229,53 @@ If you need to add a new trace type or a new algorithm, please see [here](/doc/a
 
 
 ---
-### Questions? 
+<!-- TOC --><a name="open-source-cache-traces"></a>
+## Open source cache traces
+In the [repo](/data/), there are sample (one from cloudphysics and one from twitter) traces in different formats (csv, txt, vscsi, and oracleGeneral). Note that the provided traces are **very small** samples and __should not be used for evaluating different algorithms' miss ratios__. The full traces can be found either with the original release or the processed oracleGeneral format. 
+
+Note that the oracleGeneral traces are compressed with [zstd](https://github.com/facebook/zstd) and have the following format:
+
+```
+struct {
+    uint32_t timestamp;
+    uint64_t obj_id;
+    uint32_t obj_size;
+    int64_t next_access_vtime;  // -1 if no next access
+}
+```
+The compressed traces can be used with libCacheSim without decompression. And libCacheSim provides a `tracePrint` tool to print the trace in human-readable format. 
+
+
+| Dataset       | Year |    Type   |                                      Original release                                     |                                OracleGeneral format                                |
+|---------------|------|:---------:|:-----------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------:|
+| Tencent Photo | 2018 |   object  |                      [link](http://iotta.snia.org/traces/parallel?only=27476)                     | [link](https://ftp.pdl.cmu.edu/pub/datasets/twemcacheWorkload/cacheDatasets/tencentPhoto/) |
+| WikiCDN       | 2019 |   object  |          [link](https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Caching)          |     [link](https://ftp.pdl.cmu.edu/pub/datasets/twemcacheWorkload/cacheDatasets/wiki/)     |
+| Tencent CBS   | 2020 |   block   |                      [link](http://iotta.snia.org/traces/parallel?only=27917)                     | [link](https://ftp.pdl.cmu.edu/pub/datasets/twemcacheWorkload/cacheDatasets/tencentBlock/) |
+| Alibaba CBS   | 2020 |   block   |                          [link](https://github.com/alibaba/block-traces)                          | [link](https://ftp.pdl.cmu.edu/pub/datasets/twemcacheWorkload/cacheDatasets/alibabaBlock/) |
+| Twitter       | 2020 | key-value |                          [link](https://github.com/twitter/cache-traces)                          | [link](https://ftp.pdl.cmu.edu/pub/datasets/twemcacheWorkload/cacheDatasets/twitter/)      |
+| MetaKV        | 2022 | key-value | [link](https://cachelib.org/docs/Cache_Library_User_Guides/Cachebench_FB_HW_eval/#list-of-traces) | [link](https://ftp.pdl.cmu.edu/pub/datasets/twemcacheWorkload/cacheDatasets/metaKV/)       |
+| MetaCDN       | 2023 | object    | [link](https://cachelib.org/docs/Cache_Library_User_Guides/Cachebench_FB_HW_eval/#list-of-traces) | [link](https://ftp.pdl.cmu.edu/pub/datasets/twemcacheWorkload/cacheDatasets/metaCDN/)      |
+
+Among the large number of traces, I recommend using the newer traces from Twitter (cluster52), Wiki, and Meta. 
+
+
+---
+<!-- TOC --><a name="questions"></a>
+## Questions? 
 Please join the Google group https://groups.google.com/g/libcachesim and ask questions.
 
 
 ---  
-### Contributions 
+<!-- TOC --><a name="contributions"></a>
+## Contributions 
 We gladly welcome pull requests.
 Before making any large changes, we recommend opening an issue and discussing your proposed changes.  
 If the changes are minor, then feel free to make them without discussion. 
 This project adheres to Google's coding style. By participating, you are expected to uphold this code. 
 
 ---
-### Reference
+<!-- TOC --><a name="reference"></a>
+## Reference
 ```
 @inproceedings{yang2020-workload,
     author = {Juncheng Yang and Yao Yue and K. V. Rashmi},
@@ -245,9 +316,12 @@ If you used libCacheSim in your research, please cite the above papers. And we w
 ---
 
 
-### License
+<!-- TOC --><a name="license"></a>
+## License
 See [LICENSE](LICENSE) for details.
 
-### Related
+<!-- TOC --><a name="related"></a>
+## Related
 * [PyMimircache](https://github.com/1a1a11a/PyMimircache): a python based cache trace analysis platform, now deprecated
 ---
+
diff --git a/data/trace.csv → data/cloudphyicsIO.csv b/data/trace.csv → data/cloudphyicsIO.csv
diff --git a/data/trace.oracleGeneral.bin → data/cloudphyicsIO.oracleGeneral.bin b/data/trace.oracleGeneral.bin → data/cloudphyicsIO.oracleGeneral.bin
diff --git a/data/trace.txt → data/cloudphyicsIO.txt b/data/trace.txt → data/cloudphyicsIO.txt
diff --git a/data/trace.vscsi → data/cloudphyicsIO.vscsi b/data/trace.vscsi → data/cloudphyicsIO.vscsi