Page Comparison

This is a basic tutorial for using the Gavagai Explorer API.

...

1. Get an account with Gavagai Explorer at explorer.gavagai.se if you haven’t already done so. For using the Gavagai Explorer API, you will need your login credentials (email address and password).
2. Test that the account is setup and working properly by performing a call to the following method. 

   https://ethersourceapi.gavagai.se/explorer/v1/projects
   Curl example:
   curl -u username:password "https://ethersourceapi.gavagai.se/explorer/v1/projects"

3. This should give you the following response, as you haven’t created any projects yet (if you already did create some projects, you should get an overview):
   []

...

We will use the following request to create a new project:

https://ethersourceapi.gavagai.se/explorer/v1/projects

1. Download the csv file here: tripadvisor-hotel-monaco-san-francisco-100.csv (This is a shortened version of the Hotel Monaco TripAdvisor example on our website.)

2. Send a POST request to the /projects endpoint using the csv file in the body and "Content-Type: multipart/form-data" in the header. You can use the "title" query parameter to set a title for your project.
   Curl example:
   curl -X POST -u username:password -H "Content-Type: multipart/form-data" -F "file=@/mypath/tripadvisor-hotel-monaco-san-francisco-100.csv" "https://ethersourceapi.gavagai.se/explorer/v1/projects?title=Hotel%20Monaco"

3. This should give you a response in form of the id of the project that you just created, for example: {"id":5374}

4. Now send a GET request to the /projects endpoint to see some information about your recently created project (as well as all the other projects your account owns). To see information about a specific project, append its id to your URL: https://ethersourceapi.gavagai.se/explorer/v1/projects/{id}

For more information, you can check out the API documentation for /projects.

...

To explore a project, you will send a request to the following URL:

https://ethersourceapi.gavagai.se/explorer/v1/projects/{id}/explore

You will also need to supply a ProjectExplorationContext containing several different properties that are needed for exploration. The context definition has to be supplied in JSON format. You can learn more about these formats here.

{
  "columnHeaderId" : 12345,
  "language" : "...",
  "topicsPageSize" : 12345,
  "associationsPageSize" : 12345,
  "ignoreTerms" : [ "...", "..." ],
  "filters" : [ {
    "value" : "...",
    "operator" : "NOT_EQUAL",
    "columnHeaderId" : 12345
  }, {
    "value" : "...",
    "operator" : "EQUALS",
    "columnHeaderId" : 12345
  } ],
  "groups" : [ {
    "title" : "...",
    "topics" : [ {
      "title" : "...",
      "terms" : [ "...", "..." ]
    }, {
      "title" : "...",
      "terms" : [ "...", "..." ]
    } ],
    "pinned" : true
  }, {
    "title" : "...",
    "topics" : [ {
      "title" : "...",
      "terms" : [ "...", "..." ]
    }, {
      "title" : "...",
      "terms" : [ "...", "..." ]
    } ],
    "pinned" : true
  } ],
  "acceptOverageToken" : "...",
  "conceptFilter" : {
    "inclusive" : true,
    "filtered" : true,
    "conceptId" : 12345
  }
}

Some of the attributes in the ProjectExplorationContext are optional, others are required. For this tutorial, we will set the attributes columnHeaderId, language, topicsPageSize, ignoreTerms, and groups (of which the first two are mandatory).

1. Perform a GET request to  https://ethersourceapi.gavagai.se/explorer/v1/projects. Look at the recently created project and note down the relevant attribute values from its headers array. We want to explore the header "review", so its id is the columnHeaderId needed. Here we can also note down the language (specified according to ISO 639-1) from the suggestedLanguage attribute in case we don't already know what language our texts are written in.
2. Next is the topicsPageSize. This parameter affects the number of topics that will be retrieved in each exploration. By default, this number is 30, but let's say we want to see the top 40 topics this time. 
3. Let's assume we already know that "hotel" is not a very informative term for exploring hotel reviews (all the reviews should be about this particular hotel, after all). So we add it together with its morphological variant "hotels" to the list of ignoreTerms.
4. Finally, we want to specify a couple of topics and terms. These are always organised in groups, and each group has to contain at least one topic. Let's add one group called "staff" with a topic titled "staff" that contains the terms "staff", "the staff", "general staff", as well as the topic "manager", containing "manager" and "the manager". Add another group titled "stay" with a topic "stay" and the terms "stay", "stayed", and "staying". We also want to pin the groups ("pinned": true) to ensure they will be noticeable.
5. Our final ProjectExplorationContext will look like as follows:
   

   {
     "columnHeaderId" : your_id,
     "language" : "EN",
     "topicsPageSize" : 40,
     "ignoreTerms" : [ "hotel", "hotels" ],
     "groups" : [ {
       "title" : "staff",
       "topics" : [ {
         "title" : "staff",
         "terms" : [ "staff", "the staff", "general staff" ]
       }, {
         "title" : "manager",
         "terms" : [ "manager", "the manager" ]
       } ],
       "pinned" : true
     }, {
       "title" : "stay",
       "topics" : [ {
         "title" : "stay",
         "terms" : [ "stay", "stayed", "staying" ]
       } ],
       "pinned" : true
     } ]
   }

6. Send a POST request to the /explore endpoint that contains your exploration context as body and "Content-Type: application/json;charset=UTF-8" as header.
   Curl example:
   curl -X POST -u username:password -H "Content-Type: application/json;charset=UTF-8" -d '{
  "columnHeaderId" : your_id,
  "language" : "EN",
  "topicsPageSize" : 40,
  "ignoreTerms" : [ "hotel", "hotels" ],
  "groups" : [ {
    "title" : "staff",
    "topics" : [ {
      "title" : "staff",
      "terms" : [ "staff", "the staff", "general staff" ]
    }, {
    "title" : "manager",
    "terms" : [ "manager", "the manager" ]
    } ],
    "pinned" : true
  }, {
    "title" : "stay",
    "topics" : [ {
       "title" : "stay",
       "terms" : [ "stay", "stayed", "staying" ]
     } ],
    "pinned" : true
  } ]
}' "https://ethersourceapi.gavagai.se/explorer/v1/projects/{id}/explore"

7. Now, check your project and see for yourself if your project was explored by sending a GET request to https://ethersourceapi.gavagai.se/explorer/v1/projects/{id}/explore.

For more information, you can check out the API documentation for projects/{id}/explore.

...

In this part of the tutorial we will make a request to the /append endpoint.

https://ethersourceapi.gavagai.se/explorer/v1/projects/{id}/append

1. Download the following csv file: tripadvisor-hotel-monaco-san-francisco-30-append.csv (This is another shortened version of the Hotel Monaco TripAdvisor example on our website, containing the last 30 texts.)

2. Send a POST request to the /append endpoint using the csv file in the body and "Content-Type: multipart/form-data" in the header. Use your project id as a url parameter.
   Curl example:
   curl -X POST -u username:password -H "Content-Type: multipart/form-data" -F "file=@/mypath/tripadvisor-hotel-monaco-san-francisco-30-append.csv" "https://ethersourceapi.gavagai.se/explorer/projects/{id}/append

3. This should give you a response in form of the id of the project that you just appended to, for example: {"id":5374}

4. Now send a GET request to the /projects endpoint (with the project id as a url parameter) endpoint to confirm the number of rows (which should be 130).

5. To make the changes show in your exploration, send another POST request to the /explore endpoint (see III. Exploring a project99385403).

For more information, you can check out the API documentation for /append (form data) and /append (json).

...

If your account does not have sufficient credits to explore a project, you will have to accept overage to be able to explore (please be aware that this will result in a charge to your credit card). To accept overage (and charging your card), you will have to include an extra step when sending requests to the /explore endpoint (see III. Exploring a project99385403 for exploring without accepting overage).

1. After creating your project (see II. Creating a project99385403), send a GET request to https://ethersourceapi.gavagai.se/explorer/v1/projects/{id}/explore and note the status message and state which should say "Not started" / "NOT_STARTED".
2. Send a POST request to https://ethersourceapi.gavagai.se/explorer/v1/projects/{id}/explore, as you did in III99385403. Exploring a project.
3. Send a GET request to the same endpoint to see if the project was explored or not. The response will look something like this:
   {
  "status": {
    "startTime": 0,
    "progress": 100,
    "message": "No credits left. Accept with overage token",
    "state": "NO_CREDITS",
    "acceptOverageToken": "1101e4df-e80b-4cbc-s79a-cd3d9af61511",
    "overageAmount": 213,
    "productPriceInCents": 81,
    "currency": "SEK"
  },
  "groups": [],
  "ignoreTerms": [
    "hotels",
    "hotel"
  ],
  "language": "EN",
  "columnHeaderId": 70075,
  "filters": [],
  "tones": [
    "NEGATIVITY",
    "POSITIVITY",
    "SKEPTICISM"
  ],
  "totalDocumentCount": 0
}
   
4. As you can see, the status message and code should now say "No credits left. Accept with overage token" and "NO_CREDITS" respectively. You will also see that the acceptOverageToken now has a value, and you have some additional information about the overage amount in the overageAmount, productPriceInCents, andcurrency fields. If you are okay with the amount, note down the acceptOverageToken and add it to the ProjectExplorationContext. Then, send a POST request to the /explore endpoint with your updated ProjectExplorationContext.
5. Now, check your project again to see if your project was explored by sending a GET request to https://ethersourceapi.gavagai.se/explorer/v1/projects/{id}/explore.

   
   

Related articles

...