Skip to content

Commit

Permalink
### 1.2.1
Browse files Browse the repository at this point in the history
  * Error management improvement
  * Added the possibility to do multiple concurrent request to batch API
  • Loading branch information
pvdlg committed Jan 19, 2014
1 parent a066c37 commit ea5b622
Show file tree
Hide file tree
Showing 18 changed files with 416 additions and 310 deletions.
15 changes: 10 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The easiest way to incorporate the SDK into your Java project is to use Maven. S
<dependency>
<groupId>com.syncthemall</groupId>
<artifactId>diffbot-java-sdk</artifactId>
<version>1.2.0</version>
<version>1.2.1</version>
</dependency>
```

Expand Down Expand Up @@ -116,7 +116,6 @@ Future<Products> fProducts = api.products().analyze("<web page URL>").queue();
Future<Classified> fClassified = api.classified().analyze("<web page URL>").queue();
```

Note that this can be done concurrently by multiple threads. The Diffbot class is fully thread-safe.
At this point no call has been done to Diffbot. To obtain the results:
```java
Article article = fArticle.get();
Expand All @@ -126,20 +125,26 @@ Products products = fProducts.get();
Classified classified = fClassified.get();
```

The first line trigger a batch request that retrieves the results for all the requests added since the last call to `Future#get()`. The second line doesn't need to do any API call as the result was retrieve during the
execution of the fist line.
The first line trigger one or more batch request (depending the concurrentBatchRequest value) that retrieves the results for up to (concurrentBatchRequest * maxBatchRequest) requests added since the last call to `Future#get()`.
The second line doesn't need to do any API call if the result was retrieved during the execution of the fist line.

Note that this can be done concurrently by multiple threads. The Diffbot class is fully thread-safe.

Running the JUnit tests
-----------------------
In order to run the JUnit test add to your Maven settings.xml
```xml
<properties>
<diffbot.key>0813abb2bfc05b5ea878e68848861259</diffbot.key>
<diffbot.key><your API key></diffbot.key>
</properties>
```

Change log
----------
### 1.2.1
* Error management improvement
* Added the possibility to do multiple concurrent request to batch API

### 1.2.0
* Added Image API
* Added Product API
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<groupId>com.syncthemall</groupId>
<artifactId>diffbot-java-sdk</artifactId>
<packaging>jar</packaging>
<version>1.2.0</version>
<version>1.2.1</version>
<name>diffbot-java-sdk</name>
<inceptionYear>2013</inceptionYear>
<url>https://github.com/vanduynslagerp/diffbot-java-sdk</url>
Expand Down
17 changes: 1 addition & 16 deletions src/main/java/com/syncthemall/diffbot/Constants.java
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
public final class Constants {

/** User agent. Written in the header of the generated HTML. */
public static final String USER_AGENT = "diffbot-java-sdk/1.2.0";
public static final String USER_AGENT = "diffbot-java-sdk/1.2.1";

/** Batch API URL */
public static final String BATCH_URL = "http://www.diffbot.com/api/batch";
Expand Down Expand Up @@ -71,21 +71,6 @@ public final class Constants {
/** The attribute {@code stats}. */
public static final String STATS = "stats";

/** The attribute {@code "application/json"}. */
public static final String APPLICATION_JSON = "application/json";

/** The attribute {@code Content-Type}. */
public static final String CONTENT_TYPE = "Content-Type";

/** The attribute {@code statusCode}. */
public static final String STATUS_CODE = "statusCode";

/** The attribute {@code message}. */
public static final String MESSAGE = "message";

/** The attribute {@code errorCode}. */
public static final String ERROR_CODE = "errorCode";

/** The attribute {@code error}. */
public static final String ERROR = "error";

Expand Down
98 changes: 75 additions & 23 deletions src/main/java/com/syncthemall/diffbot/Diffbot.java
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
import static com.syncthemall.diffbot.Constants.USER_AGENT;

import java.io.IOException;
import java.text.MessageFormat;
import java.util.ArrayList;
import java.util.List;
import java.util.ResourceBundle;
Expand Down Expand Up @@ -142,8 +143,6 @@
* Future&lt;Classified&gt; fClassified = diffbot.classifier().analyze(&quot;web page URL&quot;).queue();
* </pre>
* <p>
* Note that this can be done concurrently by multiple threads. The {@code Diffbot} class is fully thread-safe.
* <p>
* At this point no call has been done to Diffbot. To obtain the results:
*
* <pre>
Expand All @@ -154,9 +153,12 @@
* Classified classified = fClassified.get();
* </pre>
*
* The first line trigger a batch request that retrieves the results for all the requests added since the last call to
* {@code Future#get()}. The second line doesn't need to do any API call as the result was retrieve during the execution
* of the fist line.
* The first line trigger one or more batch request (depending the concurrentBatchRequest value) that retrieves the
* results for up to {@code concurrentBatchRequest * maxBatchRequest} requests added since the last call to
* {@code Future#get()}. The second line doesn't need to do any API call if the result was retrieved during the
* execution of the fist line.
* <p>
* Note that this can be done concurrently by multiple threads. The {@code Diffbot} class is fully thread-safe.
*
* @author Pierre-Denis Vanduynslager <[email protected]>
*/
Expand All @@ -169,6 +171,7 @@ public class Diffbot {
private List<Future<? extends Model>> futures = new ArrayList<Future<? extends Model>>();
private int maxBatchRequest = 25;
private int batchRequestTimeout = 300000;
private int concurrentBatchRequest = 1;

/**
* Constructor allowing to set the Diffbot developer token and the underlying implementations for the HTTP client
Expand Down Expand Up @@ -301,7 +304,9 @@ public class Analyze extends DiffbotRequest<com.syncthemall.diffbot.model.frontp
*/
protected Analyze(final String url) {
super(Diffbot.this, com.syncthemall.diffbot.model.frontpage.Frontpage.class, FRONTPAGE_URI);
set(URL, url);
set(URL,
com.google.api.client.util.Preconditions.checkNotNull(url,
MessageFormat.format(bundle.getString("required.parameter.null"), URL)));
set(FORMAT, "xml");
}

Expand Down Expand Up @@ -365,7 +370,9 @@ public class Analyze extends DiffbotRequest<com.syncthemall.diffbot.model.articl
*/
protected Analyze(final String url) {
super(Diffbot.this, com.syncthemall.diffbot.model.article.Article.class, ARTICLE_URI);
set(URL, url);
set(URL,
com.google.api.client.util.Preconditions.checkNotNull(url,
MessageFormat.format(bundle.getString("required.parameter.null"), URL)));
}

/**
Expand All @@ -380,7 +387,9 @@ protected Analyze(final String url) {
* @see <a href="http://www.diffbot.com/dev/docs/article">Diffbot - Article API Documentation</a>
*/
public final Analyze withFields(final String fields) {
set(FIELDS, fields);
set(FIELDS,
com.google.api.client.util.Preconditions.checkNotNull(fields,
MessageFormat.format(bundle.getString("required.parameter.null"), FIELDS)));
return this;
}

Expand All @@ -392,7 +401,9 @@ public final Analyze withFields(final String fields) {
* @return the instance of this request
*/
public final Analyze withTimeout(final int timeout) {
set(TIMEOUT, String.valueOf(timeout));
set(TIMEOUT,
com.google.api.client.util.Preconditions.checkNotNull(timeout,
MessageFormat.format(bundle.getString("required.parameter.null"), TIMEOUT)));
setReadTimeout(timeout);
return this;
}
Expand Down Expand Up @@ -457,7 +468,9 @@ public class Analyze extends DiffbotRequest<com.syncthemall.diffbot.model.images
*/
protected Analyze(final String url) {
super(Diffbot.this, com.syncthemall.diffbot.model.images.Images.class, IMAGE_URI);
set(URL, url);
set(URL,
com.google.api.client.util.Preconditions.checkNotNull(url,
MessageFormat.format(bundle.getString("required.parameter.null"), URL)));
}

/**
Expand All @@ -472,7 +485,9 @@ protected Analyze(final String url) {
* @see <a href="http://www.diffbot.com/dev/docs/image">Diffbot - Image API Documentation</a>
*/
public final Analyze withFields(final String fields) {
set(FIELDS, fields);
set(FIELDS,
com.google.api.client.util.Preconditions.checkNotNull(fields,
MessageFormat.format(bundle.getString("required.parameter.null"), FIELDS)));
return this;
}

Expand All @@ -484,7 +499,9 @@ public final Analyze withFields(final String fields) {
* @return the instance of this request
*/
public final Analyze withTimeout(final int timeout) {
set(TIMEOUT, String.valueOf(timeout));
set(TIMEOUT,
com.google.api.client.util.Preconditions.checkNotNull(timeout,
MessageFormat.format(bundle.getString("required.parameter.null"), TIMEOUT)));
setReadTimeout(timeout);
return this;
}
Expand Down Expand Up @@ -549,7 +566,9 @@ public class Analyze extends DiffbotRequest<com.syncthemall.diffbot.model.produc
*/
protected Analyze(final String url) {
super(Diffbot.this, com.syncthemall.diffbot.model.products.Products.class, PRODUCT_URI);
set(URL, url);
set(URL,
com.google.api.client.util.Preconditions.checkNotNull(url,
MessageFormat.format(bundle.getString("required.parameter.null"), URL)));
}

/**
Expand All @@ -564,7 +583,9 @@ protected Analyze(final String url) {
* @see <a href="http://www.diffbot.com/dev/docs/product">Diffbot - Product API Documentation</a>
*/
public final Analyze withFields(final String fields) {
set(FIELDS, fields);
set(FIELDS,
com.google.api.client.util.Preconditions.checkNotNull(fields,
MessageFormat.format(bundle.getString("required.parameter.null"), FIELDS)));
return this;
}

Expand All @@ -576,7 +597,9 @@ public final Analyze withFields(final String fields) {
* @return the instance of this request
*/
public final Analyze withTimeout(final int timeout) {
set(TIMEOUT, String.valueOf(timeout));
set(TIMEOUT,
com.google.api.client.util.Preconditions.checkNotNull(timeout,
MessageFormat.format(bundle.getString("required.parameter.null"), TIMEOUT)));
setReadTimeout(timeout);
return this;
}
Expand Down Expand Up @@ -641,7 +664,9 @@ public class Analyze extends DiffbotRequest<com.syncthemall.diffbot.model.classi
*/
protected Analyze(final String url) {
super(Diffbot.this, com.syncthemall.diffbot.model.classifier.Classified.class, CLASSIFIER_URI);
set(URL, url);
set(URL,
com.google.api.client.util.Preconditions.checkNotNull(url,
MessageFormat.format(bundle.getString("required.parameter.null"), URL)));
}

/**
Expand All @@ -656,7 +681,9 @@ protected Analyze(final String url) {
* @see <a href="http://www.diffbot.com/dev/docs/product">Diffbot - Product API Documentation</a>
*/
public final Analyze withFields(final String fields) {
set(FIELDS, fields);
set(FIELDS,
com.google.api.client.util.Preconditions.checkNotNull(fields,
MessageFormat.format(bundle.getString("required.parameter.null"), FIELDS)));
return this;
}

Expand All @@ -668,7 +695,9 @@ public final Analyze withFields(final String fields) {
* @return the instance of this request
*/
public final Analyze withTimeout(final int timeout) {
set(TIMEOUT, String.valueOf(timeout));
set(TIMEOUT,
com.google.api.client.util.Preconditions.checkNotNull(timeout,
MessageFormat.format(bundle.getString("required.parameter.null"), TIMEOUT)));
setReadTimeout(timeout);
return this;
}
Expand All @@ -683,7 +712,9 @@ public final Analyze withTimeout(final int timeout) {
* @return the instance of this request
*/
public final Analyze withMode(final PageType type) {
set(MODE, type.getKey());
set(MODE,
com.google.api.client.util.Preconditions.checkNotNull(type.getKey(),
MessageFormat.format(bundle.getString("required.parameter.null"), MODE)));
return this;
}

Expand Down Expand Up @@ -722,10 +753,7 @@ public final void setMaxBatchRequest(final int maxBatchRequest) {
this.maxBatchRequest = maxBatchRequest;
}

/**
* @return the timeout of a batch request
*/
public final int getBatchRequestTimeout() {
protected final int getBatchRequestTimeout() {
return batchRequestTimeout;
}

Expand All @@ -737,4 +765,28 @@ public final void setBatchRequestTimeout(final int batchRequestTimeout) {
this.batchRequestTimeout = batchRequestTimeout;
}

protected final int getConcurrentBatchRequest() {
return concurrentBatchRequest;
}

/**
* Every time {@link Future#get()} is called, the Diffbot batch API will be requested with up to
* {@code maxBatchRequest} sub-requests and up to {@code concurrentBatchRequest} times.<br>
* <p>
* ie : if {@code concurrentBatchRequest} is set to 3 and {@code maxBatchRequest} to 25 and there is 100 sub-request
* in the queue (added with {@link DiffbotRequest#queue()}) then 3 asynchronous http calls will be made to the
* Diffbot API, each one containing 25 sub-requests.<br>
* A total of 75 sub-request will be executed, including the one for the {@link Future#get()} initiator call. The
* remaining 25 sub-request in the queue will be executed when a call to {@link Future#get()} will be made for one
* of them.
*
* @param concurrentBatchRequest maximum number of concurrent batch requests (minimum 1)
*/
public final void setConcurrentBatchRequest(final int concurrentBatchRequest) {
if (maxBatchRequest < 1) {
throw new IllegalArgumentException(bundle.getString("max.concurrent.batch"));
}
this.concurrentBatchRequest = concurrentBatchRequest;
}

}
Loading

0 comments on commit ea5b622

Please sign in to comment.