Skip to content

Home

Jump to bottom Edit New page
Nadeem Mohammad edited this page Sep 5, 2016 · 32 revisions

Dexecutor is a very light weight framework to execute dependent/Independent tasks in a reliable way, to do this it provides the minimal API.

  1. An API to add nodes in the graph (addDependency, addIndependent, addAsDependentOnAllLeafNodes, addAsDependencyToAllInitialNodes Later two are the hybrid version of the first two)
  2. and the other to execute the nodes in order.

Install Dexecutor

Maven

<dependency>
  <groupId>com.github.dexecutor</groupId>
  <artifactId>dexecutor-core</artifactId>
  <version>LATEST_RELEASE</version>
</dependency>

Refer this for other options/build tools

Usage

Lets take a look at an example, here is the content of DefaultDependentTasksExecutorIntegrationTest, which would help you understand the API

@Test
public void testDependentTaskExecution() {
	ExecutorService executorService = newExecutor();

	try {
		DefaultDependentTasksExecutor<Integer, Integer> executor = newTaskExecutor(executorService);

		executor.addDependency(1, 2);
		executor.addDependency(1, 2);
		executor.addDependency(1, 3);
		executor.addDependency(3, 4);
		executor.addDependency(3, 5);
		executor.addDependency(3, 6);
		//executor.addDependency(10, 2); // cycle
		executor.addDependency(2, 7);
		executor.addDependency(2, 9);
		executor.addDependency(2, 8);
		executor.addDependency(9, 10);
		executor.addDependency(12, 13);
		executor.addDependency(13, 4);
		executor.addDependency(13, 14);
		executor.addIndependent(11);

		StringWriter writer = new StringWriter();
		executor.print(writer);
		System.out.println(writer);

		executor.execute(ExecutionBehavior.RETRY_ONCE_TERMINATING);
		System.out.println("*** Done ***");
	} finally {
		try {
			executorService.shutdownNow();
			executorService.awaitTermination(1, TimeUnit.SECONDS);
		} catch (InterruptedException e) {
			
		}
	}
}

private ExecutorService newExecutor() {
	return Executors.newFixedThreadPool(ThreadPoolUtil.ioIntesivePoolSize());
}

private DefaultDependentTasksExecutor<Integer, Integer> newTaskExecutor(ExecutorService executorService) {
	return new DefaultDependentTasksExecutor<Integer, Integer>(executorService, new SleepyTaskProvider());
}

private static class SleepyTaskProvider implements TaskProvider<Integer, Integer> {

	public Task<Integer, Integer> provid(final Integer id) {

		return new Task<Integer, Integer>() {

			public Integer execute() {
				try {
					Thread.sleep(500);
				} catch (InterruptedException e) {
					e.printStackTrace();
				}
				return id;
			}
		};			
	}		
}

As can be seen above, DefaultDependentTasksExecutorIntegrationTest requires two things

  1. Instance of ExecutorService : API would use this Service to schedule tasks
  2. Instance of TaskProvider : API represents graph using just the basic information (could be task id), it consults the TaskProvider to provide the task when it comes to actual execution.

##There are two phases

  1. Graph construction: When you say executor.addDependency(1, 2) it means tasks 1 should finish before task 2can start, executor.addIndependent(11) means as of now 11 does not depended on other nodes, but later time dependency to it can be added.
    executor.addDependency(1, 2);
    executor.addDependency(1, 3);
    executor.addDependency(3, 4);
    executor.addDependency(3, 5);
    executor.addDependency(3, 6);
    //executor.addDependency(10, 2); // cycle
    executor.addDependency(2, 7);
    executor.addDependency(2, 9);
    executor.addDependency(2, 8);
    executor.addDependency(9, 10);
    executor.addDependency(12, 13);
    executor.addDependency(13, 4);
    executor.addDependency(13, 14);
    executor.addIndependent(11);

Which would generate the following graph

dexecutor-graph.png

  1. Tasks execution

    executor.execute(ExecutionBehavior.RETRY_ONCE_TERMINATING);

Console Output

 21:18:02.080 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 1 node
 21:18:02.089 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 11 node
 21:18:02.089 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 12 node
 21:18:02.091 [pool-1-thread-2] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 11
 21:18:02.091 [pool-1-thread-3] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 12
 21:18:02.091 [pool-1-thread-1] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 1
 21:18:02.592 [pool-1-thread-1] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 1, Execution Done!
 21:18:02.594 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 1 done
 21:18:02.594 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 2 node
 21:18:02.594 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 3 node
 21:18:02.595 [pool-1-thread-2] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 11, Execution Done!
 21:18:02.595 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 11 done
 21:18:02.597 [pool-1-thread-5] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 3
 21:18:02.599 [pool-1-thread-3] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 12, Execution Done!
 21:18:02.599 [pool-1-thread-4] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 2
 21:18:02.599 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 12 done
 21:18:02.599 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 13 node
 21:18:02.600 [pool-1-thread-6] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 13
 21:18:03.099 [pool-1-thread-4] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 2, Execution Done!
 21:18:03.099 [pool-1-thread-5] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 3, Execution Done!
 21:18:03.099 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 2 done
 21:18:03.099 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 7 node
 21:18:03.099 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 9 node
 21:18:03.100 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 8 node
 21:18:03.100 [pool-1-thread-7] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 7
 21:18:03.100 [pool-1-thread-8] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 9
 21:18:03.100 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 3 done
 21:18:03.101 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 159 - node 4 depends on [3, 13]
 21:18:03.101 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 5 node
 21:18:03.101 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 6 node
 21:18:03.102 [pool-1-thread-11] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 6
 21:18:03.103 [pool-1-thread-9] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 8
 21:18:03.104 [pool-1-thread-6] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 13, Execution Done!
 21:18:03.105 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 13 done
 21:18:03.105 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 4 node
 21:18:03.106 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 14 node
 21:18:03.107 [pool-1-thread-10] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 5
 21:18:03.108 [pool-1-thread-13] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 14
 21:18:03.109 [pool-1-thread-12] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 4
 21:18:03.600 [pool-1-thread-7] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 7, Execution Done!
 21:18:03.601 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 7 done
 21:18:03.602 [pool-1-thread-8] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 9, Execution Done!
 21:18:03.603 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 9 done
 21:18:03.603 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doExecute 155 - Going to schedule 10 node
 21:18:03.604 [pool-1-thread-14] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 288 - Executing Node # 10
 21:18:03.605 [pool-1-thread-9] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 8, Execution Done!
 21:18:03.606 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 8 done
 21:18:03.607 [pool-1-thread-11] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 6, Execution Done!
 21:18:03.608 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 6 done
 21:18:03.609 [pool-1-thread-10] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 5, Execution Done!
 21:18:03.610 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 5 done
 21:18:03.611 [pool-1-thread-13] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 14, Execution Done!
 21:18:03.612 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 14 done
 21:18:03.613 [pool-1-thread-12] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 4, Execution Done!
 21:18:03.614 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 4 done
 21:18:04.105 [pool-1-thread-14] DEBUG c.g.d.e.DefaultDependentTasksExecutor$ExecutorTask.execute 293 - Node # 10, Execution Done!
 21:18:04.106 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.doWaitForExecution 181 - Processing of node 10 done
 21:18:04.106 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.execute 138 - Total Time taken to process 14 jobs is 2027 ms.
 21:18:04.106 [main] DEBUG c.g.d.e.DefaultDependentTasksExecutor.execute 139 - Processed Nodes Ordering [1, 11, 12, 2, 3, 13, 7, 9, 8, 6, 5, 14, 4, 10]

More Examples

Refer this project for more complex example

Add a custom footer
Add a custom sidebar
Clone this wiki locally