Create a tool that counts the size of each column family #936

jakubcech · 2018-08-13T13:43:03Z

Description

We want to figure out what is the byte size of the data in each of the database column families. The result will be a tool that does the calculation. Use on the LatestDB.

Is there an existing implementation that we can use? KeyLoad

Motivation

Help us understand what the priority on the other issues (merge columns) is.

Requirements

Structure of the database in percentiles
Length of the max entry in the column
Length of the min entry in the column
Any other interesting statistics mean bonus points for the assignee

alon-e · 2018-08-14T11:30:45Z

IOTA Related information:
https://iota.stackexchange.com/questions/798/how-to-read-the-tangle-database
https://pypi.org/project/python-rocksdb-iota/

GUI tools to manage rocksDB:

Keylord (v5.0+)
FastoNoSQL

alon-e · 2018-08-16T12:08:00Z

First Results:
on a Mainnet DB: https://dbfiles.iota.org/mainnet/1.5.3/db-667848.tar

Each Column Family size in Percent

08/16 14:44:45.442 [main] INFO  c.i.i.c.TransactionViewModelTest - ----------------------------
08/16 14:44:45.443 [main] INFO  c.i.i.c.TransactionViewModelTest - transactionBytes: 84.62 % (max values: 1)
08/16 14:44:45.443 [main] INFO  c.i.i.c.TransactionViewModelTest - addressBytes: 2.25 % (max values: 4657237)
08/16 14:44:45.443 [main] INFO  c.i.i.c.TransactionViewModelTest - tagBytes: 2.12 %  (max values: 5377047)
08/16 14:44:45.443 [main] INFO  c.i.i.c.TransactionViewModelTest - approveeBytes: 5.49 %  (max values: 44447)
08/16 14:44:45.443 [main] INFO  c.i.i.c.TransactionViewModelTest - bundleBytes: 5.49 %  (max values: 44447)
08/16 14:44:45.443 [main] INFO  c.i.i.c.TransactionViewModelTest - milestoneBytes: 0.01 % (max values: 1)
08/16 14:44:45.443 [main] INFO  c.i.i.c.TransactionViewModelTest - stateDiffBytes: 0.02 % (max values: 792)
08/16 14:44:45.443 [main] INFO  c.i.i.c.TransactionViewModelTest - Total (uncompressed): 58.18 GB

alon-e · 2018-08-16T12:08:49Z

Crude code for future use:

    @Test
    public void profileDBSectionsSize() throws Exception {

        RocksDBPersistenceProvider localRocksDBPP = new RocksDBPersistenceProvider("mainnetdb",
                "mainnetdb.log",1000);
        Tangle localTangle = new Tangle();
        localTangle.addPersistenceProvider(localRocksDBPP);
        localTangle.init();

        long counter;
        long size;
        List<Long> sizes = new LinkedList<>();

        //scan the whole DB to get the size of all the components:
        TransactionViewModel tx = TransactionViewModel.first(localTangle);
        counter = 0;
        while (tx != null) {
            if (++counter % 10000 == 0) {
                log.info("Scanned {} Transactions", counter);
            }
            tx = tx.next(localTangle);
        }
        // Key + Value + Metadata
        sizes.add(counter * (Hash.SIZE_IN_BYTES + Transaction.SIZE + 450));

        AddressViewModel add = AddressViewModel.first(localTangle);
        counter = 0;
        size = 0;
        while (add != null) {
            if (++counter % 10000 == 0) {
                log.info("Scanned {} Addresses", counter);
            }
            size += add.size();
            add = add.next(localTangle);
        }
        // Key + # of entries in each value ( + delimiter)
        sizes.add(counter * Hash.SIZE_IN_BYTES + size * (Hash.SIZE_IN_BYTES + 1));


        TagViewModel tag = TagViewModel.first(localTangle);
        counter = 0;
        size = 0;
        while (tag != null) {
            if (++counter % 10000 == 0) {
                log.info("Scanned {} Tags", counter);
            }
            size += tag.size();
            tag = tag.next(localTangle);
        }
        // Key + # of entries in each value ( + delimiter)
        sizes.add(counter * Hash.SIZE_IN_BYTES + size * (Hash.SIZE_IN_BYTES + 1));


        ApproveeViewModel approvee = ApproveeViewModel.first(localTangle);
        counter = 0;
        size = 0;
        while (approvee != null) {
            if (++counter % 10000 == 0) {
                log.info("Scanned {} Approvees", counter);
            }
            size += approvee.size();
            approvee = approvee.next(localTangle);
        }
        // Key + # of entries in each value ( + delimiter)
        sizes.add(counter * Hash.SIZE_IN_BYTES + size * (Hash.SIZE_IN_BYTES + 1));


        ApproveeViewModel bundle = ApproveeViewModel.first(localTangle);
        counter = 0;
        size = 0;
        while (bundle != null) {
            if (++counter % 10000 == 0) {
                log.info("Scanned {} Bundles", counter);
            }
            size += bundle.size();
            bundle = bundle.next(localTangle);
        }
        // Key + # of entries in each value ( + delimiter)
        sizes.add(counter * Hash.SIZE_IN_BYTES + size * (Hash.SIZE_IN_BYTES + 1));


        MilestoneViewModel milestone = MilestoneViewModel.first(localTangle);
        counter = 0;
        while (milestone != null) {
            if (++counter % 10000 == 0) {
                log.info("Scanned {} Bundles", counter);
            }
            milestone = milestone.next(localTangle);
        }
        sizes.add(counter * (Integer.BYTES + Hash.SIZE_IN_BYTES));


        MilestoneViewModel milestoneForStateDiff = MilestoneViewModel.first(localTangle);
        counter = 0;
        size = 0;
        while (milestoneForStateDiff != null) {
            if (++counter % 10000 == 0) {
                log.info("Scanned {} StateDiffs", counter);
            }
            StateDiffViewModel stateDiff = StateDiffViewModel.load(localTangle, milestoneForStateDiff.getHash());
            size += stateDiff.getDiff().size();
            milestoneForStateDiff = milestoneForStateDiff.next(localTangle);
        }

        sizes.add(counter * (Long.BYTES + Hash.SIZE_IN_BYTES) + size * (Long.BYTES + Hash.SIZE_IN_BYTES));

        double sum = sizes.stream().reduce((a,b)->a+b).get();
        int i = 0;
        log.info("----------------------------");
        log.info(String.format("transactionBytes: %.2f",  sizes.get(i++) / sum * 100));
        log.info(String.format("addressBytes: %.2f",  sizes.get(i++) / sum * 100));
        log.info(String.format("tagBytes: %.2f",  sizes.get(i++) / sum * 100));
        log.info(String.format("approveeBytes: %.2f",  sizes.get(i++) / sum * 100));
        log.info(String.format("bundleBytes: %.2f",  sizes.get(i++) / sum * 100));
        log.info(String.format("milestoneBytes: %.2f",  sizes.get(i++) / sum * 100));
        log.info(String.format("stateDiffBytes: %.2f",  sizes.get(i++) / sum * 100));
        log.info(String.format("Total (uncompressed): %.2f GB",  sum / 1073741824 /*GB */));

    }

alon-e · 2018-08-16T12:18:54Z

So my conclusion for deleting transaction is:

deleting the Transaction and its metadata covers the majority of DB size gain - > 80%.
The approvees can also be easily deleted, b/c it's indexed by the hash on the TX, and all the values can be removed, approvees add 5% to size gain.
Tags, Addresses, and Bundles (~3-5% each) require extensive work: 1. get a specific Tx field, and then read an entry, remove the Hash from the Set and store it again. I think this would cost much more than we gain - cost in I/O , Mem & Comp.
3.a. we could do this lazy somehow, by marking the key as needing a clean-up, and when itis fetched actually execute the clean-up, but this is too complicated to start w/ imo.
if we choose not to remove (3.) from the DB, then logic that iterates over these merge columns should be reviewed to see that no logic errors are introduced by missing Transaction corresponding to Hashes.

bottom line: I think we should start w/ only (1.) deleting the Transaction and attending to (4.)

GalRogozinski · 2018-10-23T08:28:18Z

Any objections to closing this?

jakubcech · 2018-10-23T08:29:54Z

None, it was left open before as @alon-e wanted to put the tool for this in a repo. But I don't think it's a priority to do anytime soon.

jakubcech added the C-Persistence label Aug 13, 2018

jakubcech assigned alon-e Aug 13, 2018

jakubcech added this to the Raanana milestone Aug 13, 2018

jakubcech mentioned this issue Aug 23, 2018

Support for deleting items in the merge column #922

Open

jakubcech closed this as completed Oct 23, 2018

GalRogozinski mentioned this issue Jan 12, 2020

Test: Add rocksdb size profiling test #1705

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a tool that counts the size of each column family #936

Create a tool that counts the size of each column family #936

jakubcech commented Aug 13, 2018 •

edited by alon-e

Loading

alon-e commented Aug 14, 2018

alon-e commented Aug 16, 2018 •

edited

Loading

alon-e commented Aug 16, 2018

alon-e commented Aug 16, 2018

GalRogozinski commented Oct 23, 2018

jakubcech commented Oct 23, 2018

Create a tool that counts the size of each column family #936

Create a tool that counts the size of each column family #936

Comments

jakubcech commented Aug 13, 2018 • edited by alon-e Loading

Description

Motivation

Requirements

alon-e commented Aug 14, 2018

alon-e commented Aug 16, 2018 • edited Loading

alon-e commented Aug 16, 2018

alon-e commented Aug 16, 2018

GalRogozinski commented Oct 23, 2018

jakubcech commented Oct 23, 2018

jakubcech commented Aug 13, 2018 •

edited by alon-e

Loading

alon-e commented Aug 16, 2018 •

edited

Loading