Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Composite blocktypes: Basic functionality #411

Merged
merged 24 commits into from
Aug 30, 2023
Merged

Composite blocktypes: Basic functionality #411

merged 24 commits into from
Aug 30, 2023

Conversation

rhazn
Copy link
Contributor

@rhazn rhazn commented Aug 2, 2023

From https://github.com/jvalue/jayvee/tree/main/rfc/0008-builtin-blocktypes

Reference #217, out of scope for this PR:

  • model validation hints concerning composite blocks
  • unit tests
  • autogenerated docs
  • adding a GTFS/CSV composite block into the stdlib

File with composite blocks to test

composite blocktype TextFileExtractor {
    property url oftype text;

    input inputName oftype None;
    output outputName oftype File;

    block FileExtractor oftype HttpExtractor { url: url; }

	block FileTextInterpreter oftype TextFileInterpreter {}

    inputName
        ->FileExtractor
        ->FileTextInterpreter
        ->outputName;
}

composite blocktype CSVExtractor {
    property url oftype text;
    property delimiter oftype text: ',';
    property enclosing oftype text: '';
    property enclosingEscape oftype text: '';

    input inputName oftype None;
    output outputName oftype Sheet;

    block TextFileExtractor oftype TextFileExtractor { url: url; }

	block FileCSVInterpreter oftype CSVInterpreter {
		delimiter: delimiter;
		enclosing: enclosing;
		enclosingEscape: enclosingEscape;
	}

    inputName
        ->TextFileExtractor
        ->FileCSVInterpreter
        ->outputName;
}

composite blocktype CarsTableInterpreterForTesting {
	input inputName oftype Sheet;
    output outputName oftype Table;

	block CarsTableInterpreter oftype TableInterpreter {
		header: true;
		columns: [
			"name" oftype text,
			"mpg" oftype decimal,
			"cyl" oftype integer,
			"disp" oftype decimal,
			"hp" oftype integer,
			"drat" oftype decimal,
			"wt" oftype decimal,
			"qsec" oftype decimal,
			"vs" oftype integer,
			"am" oftype integer,
			"gear" oftype integer,
			"carb" oftype integer
		];
	}

	inputName->CarsTableInterpreter->outputName;
}

composite blocktype SinkTester {
	input inputName oftype Table;
    output outputName oftype None;

	block CarsLoader oftype SQLiteLoader {
		table: "Cars";
		file: "./compositeblockstest.sqlite";
	}

	inputName->CarsLoader->outputName;
}

pipeline CarsPipeline {
	CarsCSVExtractor 
	   	-> TestCompositeWithInput
		-> CarsLoader;

    block CarsCSVExtractor oftype CSVExtractor {
        url: "https://gist.githubusercontent.com/noamross/e5d3e859aa0c794be10b/raw/b999fb4425b54c63cab088c0ce2c0d6ce961a563/cars.csv";
		enclosing: '"';
    }

	block TestCompositeWithInput oftype CarsTableInterpreterForTesting {
	}

	block CarsLoader oftype SinkTester {
	}
}

For testing, using e.g., the above source code:

npm run generate && npm run build && node dist/apps/interpreter/main.js example/composite-block.jv

@rhazn rhazn force-pushed the composite-blocktypes branch from 4aad709 to 9655288 Compare August 4, 2023 15:38
Base automatically changed from dev to main August 15, 2023 14:27
@rhazn rhazn force-pushed the composite-blocktypes branch from 9655288 to 77d317e Compare August 19, 2023 15:50
@rhazn rhazn changed the base branch from main to dev August 19, 2023 15:52
@rhazn rhazn mentioned this pull request Aug 25, 2023
9 tasks
@rhazn rhazn changed the title WIP: Composite blocktypes WIP: Composite blocktypes: Basic functionality Aug 25, 2023
Base automatically changed from dev to main August 28, 2023 08:58
@rhazn rhazn changed the title WIP: Composite blocktypes: Basic functionality Composite blocktypes: Basic functionality Aug 29, 2023
@rhazn rhazn force-pushed the composite-blocktypes branch from cb90802 to d766cc2 Compare August 29, 2023 15:10
@rhazn rhazn requested a review from georg-schwarz August 29, 2023 15:40
Copy link
Member

@georg-schwarz georg-schwarz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Looks quite intuitive after the refactor to me :)

I left a few very nitpicky comments. Feel free to ignore them as most of them are quite opinionated I guess ;-)

@rhazn rhazn requested a review from georg-schwarz August 30, 2023 09:58
@rhazn rhazn merged commit 458b64e into main Aug 30, 2023
2 checks passed
@rhazn rhazn deleted the composite-blocktypes branch August 30, 2023 13:05
@github-actions github-actions bot locked and limited conversation to collaborators Aug 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants