Skip to content
This repository has been archived by the owner on Jul 7, 2021. It is now read-only.

Step names #32

Merged
merged 94 commits into from
Jun 11, 2015
Merged

Step names #32

merged 94 commits into from
Jun 11, 2015

Conversation

rodneykinney
Copy link
Member

This is a refactoring with the following goals:

  • Allow naming of steps in a pipeline. The names are used in three places: 1) to name the output file 2) to label the boxes in the visualization diagram and 3) as keys which can be used to retrieve steps from a Pipeline by name
  • Remove need to specify output.persist.XXX=true for all steps in config
  • Provide more information on how steps were executed (read from disk, computed and stored, computed only, not needed)
  • Support specification of absolute URLs when overriding output locations in config
  • Cleaner API for customizing persistence (e.g. FaultTolerantRddMapper)
  • Update and improve documentation

@schmmd @cristipp FYI

rodneykinney and others added 30 commits May 12, 2015 10:49
@rodneykinney
Setting version to 1.2.1
89a1a1b
@rodneykinney
Setting version to 1.2.2-SNAPSHOT
1de79fa
@rodneykinney
Add tmpOutput option to runOne()
aa744e1
@rodneykinney
Merge branch 'master' of https://github.com/allenai/pipeline
57bf299
@rodneykinney
Change type of output override in runOne()
58fc23f
@rodneykinney
Change runOne() signature
755baad
@rodneykinney
Cleanup
9d76549
@rodneykinney
Refactor ArtifactFactory
a72721c
@rodneykinney
Cleanup
2bad5f5
@rodneykinney
Refactor ArtifactFactory
3b7e599
@rodneykinney
Refactory ArtifactFactory
fe708de
@rodneykinney
Add S3Config factory method
bb8ba4b
@rodneykinney
ExecuteShellCommand
6f82a27
@rodneykinney
Refactor ExecuteShellCommand
1a6d2e2
@rodneykinney
Refactor ExecuteShellCommand
6525dfe
@rodneykinney
Merge branch 'master' into artifact-factory
d61fdc5
@rodneykinney
Add implicit conversions, BasicPipeline example
f6606ca
@rodneykinney
Remove path parameter from persist(). Update README
9716ef9
@rodneykinney
Improve HTML
730cf08
@rodneykinney
Reformat
98243e0
@rodneykinney
ExecutionInfo
aa92617
@rodneykinney
Rename ExecuteShellCommand -> ExternalProcess
aa228ce
@rodneykinney
TrainModel example pipeline
023c157
@rodneykinney
Add python-based model-training example pipeline
0d16b79
@rodneykinney
Reorganize ExternalProcess
b72a392
@rodneykinney
Rename Serializer/Deserializer. Update README
7caa3a0
@rodneykinney
Merge branch 'master' of https://github.com/allenai/pipeline
5a985fd
@rodneykinney
Rounded corners
6ade036
@rodneykinney
Default AWS credentials
1c7f08e
@rodneykinney
Merge branch 'master' of https://github.com/allenai/pipeline into art…
518d533 
…ifact-factory
rodneykinney
rodneykinney reviewed Jun 4, 2015
View reviewed changes
README.md
@@ -1,394 +1,193 @@
AI2 Pipeline Framework
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README needed some updating.

rodneykinney added a commit that referenced this pull request Jun 11, 2015
@rodneykinney
Merge pull request #32 from rodneykinney/step-names
ddd5f4c 
Step names
@rodneykinney rodneykinney merged commit ddd5f4c into allenai:master Jun 11, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees

@jakemannix jakemannix

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rodneykinney @jakemannix @jefeweisen