Enabling request/response logging WSO2 API Manager REST API

For debugging purposes sometimes it might be useful to add request/response logging for the REST API webapps. For that you can add the following configuration to beans.xml under <cxf:bus> element.

<cxf:features>
    <cxf:logging/>
</cxf:features>

beans.xml can be found in following path: 

<APIM-Home>/repository/deployment/server/webapps/{webapp-name}/WEB-INF

After Updating beans.xml it will look like following.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:jaxrs="http://cxf.apache.org/jaxrs" xmlns:context="http://www.springframework.org/schema/context"
       xmlns:cxf="http://cxf.apache.org/core"
       xsi:schemaLocation="http://www.springframework.org/schema/beans  http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd http://cxf.apache.org/jaxrs http://cxf.apache.org/schemas/jaxrs.xsd http://cxf.apache.org/core http://cxf.apache.org/schemas/core.xsd">
    <import resource="classpath:META-INF/cxf/cxf.xml"/>
    <context:property-placeholder/>
    <context:annotation-config/>
    <bean class="org.springframework.web.context.support.ServletContextPropertyPlaceholderConfigurer"/>
    <bean class="org.springframework.beans.factory.config.PreferencesPlaceholderConfigurer"/>
    <jaxrs:server id="services" address="/">
        <jaxrs:serviceBeans>
            <bean class="org.wso2.carbon.apimgt.rest.api.publisher.SubscriptionsApi"/>
            <bean class="org.wso2.carbon.apimgt.rest.api.publisher.ApisApi"/>
            <bean class="org.wso2.carbon.apimgt.rest.api.publisher.EnvironmentsApi"/>
            <bean class="org.wso2.carbon.apimgt.rest.api.publisher.TiersApi"/>
            <bean class="org.wso2.carbon.apimgt.rest.api.publisher.ApplicationsApi"/>
            
        </jaxrs:serviceBeans>
        <jaxrs:providers>
            <!--<bean class="org.wso2.carbon.apimgt.rest.api.util.handlers.BasicAuthenticationHandler"/>
            <bean class="org.wso2.carbon.apimgt.rest.api.util.handlers.XACMLAuthenticationHandler"/>-->

            <bean class="org.codehaus.jackson.jaxrs.JacksonJsonProvider"/>
            <bean class="org.wso2.carbon.apimgt.rest.api.util.exception.GlobalThrowableMapper" />
        </jaxrs:providers>
        <jaxrs:properties>
            <!-- This is added to catch interceptor level exceptions in GlobalThrowableMapper. -->
            <entry key="map.cxf.interceptor.fault" value="true" />
        </jaxrs:properties>
    </jaxrs:server>
    <bean id="AuthenticationInterceptor" class="org.wso2.carbon.apimgt.rest.api.util.interceptors.auth.OAuthAuthenticationInterceptor" />
    <bean id="ValidationInInterceptor" class="org.wso2.carbon.apimgt.rest.api.util.interceptors.validation.ValidationInInterceptor"/>
    <cxf:bus>
        <cxf:inInterceptors>
            <ref bean="AuthenticationInterceptor"/>
            <ref bean="ValidationInInterceptor"/>
        </cxf:inInterceptors>
        <cxf:features>
            <cxf:logging />
        </cxf:features>
    </cxf:bus>
</beans>

Example message logging you can see in server's console:

[2015-12-21 12:20:14,268]  INFO - LoggingInInterceptor Inbound Message
----------------------------
ID: 3
Address: http://127.0.0.1:9764/api/am/store/v0.9//tiers/api/Bronze
Http-Method: GET
Content-Type: 
Headers: {accept-encoding=[gzip,deflate], Authorization=[Bearer 28603e8e342069446d3cc1b93b71dc46], connection=[Keep-Alive], Content-Type=[null], host=[127.0.0.1:9764], user-agent=[Apache-HttpClient/4.1.1 (java 1.5)]}
--------------------------------------
[2015-12-21 12:20:14,342]  INFO - LoggingOutInterceptor Outbound Message
---------------------------
ID: 3
Response-Code: 200
Content-Type: application/json
Headers: {Content-Type=[application/json], Date=[Mon, 21 Dec 2015 06:50:14 GMT]}
Payload: {"unitTime":60000,"tierPlan":"FREE","tierLevel":"api","stopOnQuotaReach":true,"requestCount":1,"description":"Allows 1 request(s) per minute.","name":"Bronze","attributes":{}}
--------------------------------------

API Manager new REST API - From API Creator to Consumer

In this blog we will go through the basic steps of

1. Accessing Publisher REST API

2. Creating an API
3. Publishing an API
4. Accessing Store REST API
5. Creating an Application
6. Generate access token to that application
7. Subscribing to the API using the application
8. Invoking the API

using the new REST API.

Accessing Publisher web app

The new REST API is secured using OAuth so we need to give an access token when invoking APIs. The access tokens are created on behalf of an OAuth application. So first of all we need to create an OAuth application using the DCR endpoint to if you have not already done. Please see my previous blog post for more details about it. So I am going through those steps in brief.

Creating an OAuth application using DCR endpoint:

curl -X POST -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "Content-Type: application/json" -d @payload.json http://localhost:9763/client-registration/v0.9/register

Sample payload.json file:

{
 "callbackUrl": "www.google.lk",
 "clientName": "rest_api_publisher",
 "tokenScope": "Production",
 "owner": "admin",
 "grantType": "password refresh_token",
 "saasApp": true
}

Response:

HTTP/1.1 201 Created
Date: Sun, 20 Mar 2016 03:29:57 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: WSO2 Carbon Server

{  
   "jsonString":"{\"username\":\"admin\",\"redirect_uris\":\"www.google.lk\",\"client_name\":\"admin_rest_api_publisher\",\"grant_types\":\"urn:ietf:params:oauth:grant-type:saml2-bearer iwa:ntlm implicit refresh_token client_credentials authorization_code password\"}",
   "appOwner":null,
   "clientName":null,
   "callBackURL":"www.google.lk",
   "isSaasApplication":true,
   "clientId":"xe94b1qreLVlplHjZGwLtfrmv38a",
   "clientSecret":"fOua2cfsdTowN0apzCcJ2uIfLMYa"
}

Extract the clientId and clientSecret from above response and use it to invoke the token endpoint. You have to base64 encode clientIt:clientSecret pair and use it as the Authorization: Basic header in the next step.

Get an access token using token endpoint:

In order to create an API, you need to have apim:api_create scope. So make sure you include the apim:api_create scope in your scope parameter. Here I am requesting all scopes related to API creator and publisher

curl -k -d "grant_type=password&username=admin&password=admin&scope=apim:api_create" -H "Authorization: Basic eGU5NGIxcXJlTFZscGxIalpHd0x0ZnJtdjM4YTpmT3VhMmNmc2RUb3dOMGFwekNjSjJ1SWZMTVlh" https://127.0.0.1:8243/token -v

Reponse:

< HTTP/1.1 200 OK
< Content-Type: application/json
< Pragma: no-cache
< Cache-Control: no-store
< Date: Sun, 20 Mar 2016 15:17:15 GMT
< Transfer-Encoding: chunked
< 
* Connection #0 to host 127.0.0.1 left intact
{"scope":"apim:api_create","token_type":"Bearer","expires_in":3600,"refresh_token":"fc6f5cc8fed59ca16acfd8ace97e3861","access_token":"a651175508fafcd116488d84e53ca3de"}

Note the access token a651175508fafcd116488d84e53ca3de which we will use for our next API requests.

Creating an API

Using the access token you can now invoke the API Create API. I will refer the sample payload added in the official documentation.

curl -H "Authorization: Bearer 8f1795f3d5d9c3b4d9bbeafd2e39f437" -H "Content-Type: application/json" -X POST -d @payload.json http://127.0.0.1:9763/api/am/publisher/v0.9/apis

Sample payload.json

{
"sequences": [],
"tiers": [
        "Unlimited"
    ],
"thumbnailUrl": "/registry/resource/_system/governance/apimgt/applicationdata/icons/admin/CalculatorAPI/1.0/icon",
"visibility": "PUBLIC",
"visibleRoles": [],
"visibleTenants": [],
"cacheTimeout": 300,
"endpointConfig": "{\"production_endpoints\": {\"url\": \"https://localhost:9443/am/sample/calculator/v1/api\",\"config\": null},\"sandbox_endpoints\": {\"url\": \"https://localhost:9443/am/sample/calculator/v1/api\",\"config\": null},\"endpoint_type\": \"http\" }",
"subscriptionAvailability": null,
"subscriptionAvailableTenants": [],
"destinationStatsEnabled": "Disabled",
"apiDefinition": "{\"basePath\":\"\\/calc\\/1.0\",\"host\":\"https:\\/\\/localhost:9443\",\"paths\":{\"\\/divide\":{\"get\":{\"x-auth-type\":\"Application & Application User\",\"summary\":\"divide x by y\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"type\":\"string\",\"required\":true,\"in\":\"query\"},{\"name\":\"y\",\"type\":\"string\",\"required\":true,\"in\":\"query\"}],\"responses\":{\"200\":{}}}},\"\\/subtract\":{\"get\":{\"x-auth-type\":\"Application & Application User\",\"summary\":\"subtract y from x\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"type\":\"string\",\"required\":true,\"in\":\"query\"},{\"name\":\"y\",\"type\":\"string\",\"required\":true,\"in\":\"query\"}],\"responses\":{\"200\":{}}}},\"\\/add\":{\"get\":{\"x-auth-type\":\"Application & Application User\",\"summary\":\"add x and y\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"type\":\"string\",\"required\":true,\"in\":\"query\"},{\"name\":\"y\",\"type\":\"string\",\"required\":true,\"in\":\"query\"}],\"responses\":{\"200\":{}}}},\"\\/multiply\":{\"get\":{\"x-auth-type\":\"Application & Application User\",\"summary\":\"multiply x by y\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"type\":\"string\",\"required\":true,\"in\":\"query\"},{\"name\":\"y\",\"type\":\"string\",\"required\":true,\"in\":\"query\"}],\"responses\":{\"200\":{}}}}},\"swagger\":\"2.0\",\"info\":{\"title\":\"Calculator\",\"description\":\"Simple calculator API to perform addition, subtraction, multiplication and division.\",\"version\":\"1.0\"}}",
"responseCaching": "Disabled",
"isDefaultVersion": false,
"gatewayEnvironments": "Production and Sandbox",
"businessInformation": {
"technicalOwner": null,
"technicalOwnerEmail": null,
"businessOwner": null,
"businessOwnerEmail": null
},
"transport": [
"http",
"https"
],
"tags": [
"calculator"
],
"provider": "admin",
"version": "1.0",
"name": "CalculatorSampleAPI",
"context": "/samplecalc",
"description": "Simple calculator API to perform addition, subtraction, multiplication and division."
}

Response:

< HTTP/1.1 201 Created
< Location: http://localhost:9763/api/am/publisher/v0.9/apis/96758d8d-b121-4ae9-a752-76d138295f3e
< Date: Sat, 26 Mar 2016 07:51:47 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Server: WSO2 Carbon Server
{
 "sequences": [],
 "tiers": ["Unlimited"],
 "thumbnailUrl": null,
 "visibility": "PUBLIC",
 "visibleRoles": [],
 "visibleTenants": [],
 "cacheTimeout": 300,
 "endpointConfig": "{\"production_endpoints\": {\"url\": \"https://localhost:9443/am/sample/calculator/v1/api\",\"config\": null},\"sandbox_endpoints\": {\"url\": \"https://localhost:9443/am/sample/calculator/v1/api\",\"config\": null},\"endpoint_type\": \"http\" }",
 "subscriptionAvailability": null,
 "subscriptionAvailableTenants": [],
 "destinationStatsEnabled": "Disabled",
 "apiDefinition": "{\"basePath\":\"\\/calc\\/1.0\",\"paths\":{\"\\/divide\":{\"get\":{\"summary\":\"divide x by y\",\"x-auth-type\":\"Application & Application User\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"required\":true,\"type\":\"string\",\"in\":\"query\"},{\"name\":\"y\",\"required\":true,\"type\":\"string\",\"in\":\"query\"}],\"responses\":{\"200\":{}}}},\"\\/subtract\":{\"get\":{\"summary\":\"subtract y from x\",\"x-auth-type\":\"Application & Application User\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"required\":true,\"type\":\"string\",\"in\":\"query\"},{\"name\":\"y\",\"required\":true,\"type\":\"string\",\"in\":\"query\"}],\"responses\":{\"200\":{}}}},\"\\/add\":{\"get\":{\"summary\":\"add x and y\",\"x-auth-type\":\"Application & Application User\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"required\":true,\"type\":\"string\",\"in\":\"query\"},{\"name\":\"y\",\"required\":true,\"type\":\"string\",\"in\":\"query\"}],\"responses\":{\"200\":{}}}},\"\\/multiply\":{\"get\":{\"summary\":\"multiply x by y\",\"x-auth-type\":\"Application & Application User\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"required\":true,\"type\":\"string\",\"in\":\"query\"},{\"name\":\"y\",\"required\":true,\"type\":\"string\",\"in\":\"query\"}],\"responses\":{\"200\":{}}}}},\"host\":\"https:\\/\\/localhost:9443\",\"swagger\":\"2.0\",\"info\":{\"title\":\"Calculator\",\"description\":\"Simple calculator API to perform addition, subtraction, multiplication and division.\",\"version\":\"1.0\"}}",
 "responseCaching": "Disabled",
 "isDefaultVersion": false,
 "gatewayEnvironments": "Production and Sandbox",
 "businessInformation": {
  "technicalOwner": null,
  "technicalOwnerEmail": null,
  "businessOwner": null,
  "businessOwnerEmail": null
 },
 "transport": ["http", "https"],
 "tags": ["calculator"],
 "provider": "admin",
 "version": "1.0",
 "name": "CalculatorSampleAPI",
 "context": "/samplecalc",
 "id": "96758d8d-b121-4ae9-a752-76d138295f3e",
 "description": "Simple calculator API to perform addition, subtraction, multiplication and division.",
 "status": "CREATED"
}

So we have created an API successfully.

There are few things to note.

The first one; what does it actually give it as a response? Is it same as the payload we specify when creating an API?.
You would be able to see few additional parameters in the response which you did not specify in your request.The "Id" parameter, that is the most important one. It is a UUID which uniquely identifies the API you created. When you need to refer to this API when retrieving it/ deleting it later, you can use this id.

The second one; the Location header. Did you notice you also received a Location header as a response. Let's double check once.
That is the path the new API resource is created. You can directly use it when retrieving the API.

Get a token for apim:api_view scope:
curl -k -d "grant_type=password&username=admin&password=admin&scope=apim:api_view" -H "Authorization: Basic eGU5NGIxcXJlTFZscGxIalpHd0x0ZnJtdjM4YTpmT3VhMmNmc2RUb3dOMGFwekNjSjJ1SWZMTVlh" https://127.0.0.1:8243/token -v
Response:
{"scope":"apim:api_view","token_type":"Bearer","expires_in":3600,"refresh_token":"11cc2f0283eed21b4407bab4d9a23e8c","access_token":"62692e79399d8f08b0cde3df70989b3d"}
Invoke:
curl -H "Authorization: Bearer 62692e79399d8f08b0cde3df70989b3d" -H "Content-Type: application/json" http://localhost:9763/api/am/publisher/v0.9/apis/96758d8d-b121-4ae9-a752-76d138295f3e
You will retrieve the API you added earlier.
{
 "sequences": [],
 "tiers": ["Unlimited"],
 "thumbnailUrl": null,
 "visibility": "PUBLIC",
 "visibleRoles": [],
 "visibleTenants": [],
 "cacheTimeout": 300,
 "endpointConfig": "{\"production_endpoints\": {\"url\": \"https://localhost:9443/am/sample/calculator/v1/api\",\"config\": null},\"sandbox_endpoints\": {\"url\": \"https://localhost:9443/am/sample/calculator/v1/api\",\"config\": null},\"endpoint_type\": \"http\" }",
 "subscriptionAvailability": null,
 "subscriptionAvailableTenants": [],
 "destinationStatsEnabled": "Disabled",
 "apiDefinition": "{\"basePath\":\"\\/calc\\/1.0\",\"paths\":{\"\\/divide\":{\"get\":{\"summary\":\"divide x by y\",\"x-auth-type\":\"Application & Application User\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"required\":true,\"type\":\"string\",\"in\":\"query\"},{\"name\":\"y\",\"required\":true,\"type\":\"string\",\"in\":\"query\"}],\"responses\":{\"200\":{}}}},\"\\/subtract\":{\"get\":{\"summary\":\"subtract y from x\",\"x-auth-type\":\"Application & Application User\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"required\":true,\"type\":\"string\",\"in\":\"query\"},{\"name\":\"y\",\"required\":true,\"type\":\"string\",\"in\":\"query\"}],\"responses\":{\"200\":{}}}},\"\\/add\":{\"get\":{\"summary\":\"add x and y\",\"x-auth-type\":\"Application & Application User\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"required\":true,\"type\":\"string\",\"in\":\"query\"},{\"name\":\"y\",\"required\":true,\"type\":\"string\",\"in\":\"query\"}],\"responses\":{\"200\":{}}}},\"\\/multiply\":{\"get\":{\"summary\":\"multiply x by y\",\"x-auth-type\":\"Application & Application User\",\"x-throttling-tier\":\"Unlimited\",\"produces\":[\"application\\/json\"],\"parameters\":[{\"name\":\"x\",\"required\":true,\"type\":\"string\",\"in\":\"query\"},{\"name\":\"y\",\"required\":true,\"type\":\"string\",\"in\":\"query\"}],\"responses\":{\"200\":{}}}}},\"host\":\"https:\\/\\/localhost:9443\",\"swagger\":\"2.0\",\"info\":{\"title\":\"Calculator\",\"description\":\"Simple calculator API to perform addition, subtraction, multiplication and division.\",\"version\":\"1.0\"}}",
 "responseCaching": "Disabled",
 "isDefaultVersion": false,
 "gatewayEnvironments": "Production and Sandbox",
 "businessInformation": {
  "technicalOwner": null,
  "technicalOwnerEmail": null,
  "businessOwner": null,
  "businessOwnerEmail": null
 },
 "transport": ["http", "https"],
 "tags": ["calculator"],
 "provider": "admin",
 "version": "1.0",
 "name": "CalculatorSampleAPI",
 "context": "/samplecalc",
 "id": "96758d8d-b121-4ae9-a752-76d138295f3e",
 "description": "Simple calculator API to perform addition, subtraction, multiplication and division.",
 "status": "CREATED"
}

Lets get all APIs and check our new API is there.

Generate token with apim:api_view scope:

curl -k -d "grant_type=password&username=admin&password=admin&scope=apim:api_view" -H "Authorization: Basic eGU5NGIxcXJlTFZscGxIalpHd0x0ZnJtdjM4YTpmT3VhMmNmc2RUb3dOMGFwekNjSjJ1SWZMTVlh" https://127.0.0.1:8243/token

Token generation response:

{"scope":"apim:api_view","token_type":"Bearer","expires_in":2357,"refresh_token":"528bd157f66df1db21c046737bb819eb","access_token":"ba5dc41a32387ff1db194bdf2b5d7c5f"}

Invoke get all APIs:

curl -H "Authorization: Bearer ba5dc41a32387ff1db194bdf2b5d7c5f" http://127.0.0.1:9763/api/am/publisher/v0.9/apis

Response:

< HTTP/1.1 200 OK
< Date: Sat, 26 Mar 2016 07:56:50 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Server: WSO2 Carbon Server
<

{
 "previous": "",
 "list": [{
  "provider": "admin",
  "version": "1.0",
  "name": "CalculatorSampleAPI",
  "context": "/samplecalc",
  "id": "96758d8d-b121-4ae9-a752-76d138295f3e",
  "description": "Simple calculator API to perform addition, subtraction, multiplication and division.",
  "status": "PUBLISHED"
 }],
 "count": 1,
 "next": ""
}

Publish the API

The API is now in CREATED state. We need to change its lifecycle state to PUBLISHED in order to make it visible in the Store. For this task we use the publisher API's /change-lifecycle API, which is secured with apim:api_publish scope.

Create a token with apim:api_publish scope:

curl -k -d "grant_type=password&username=admin&password=admin&scope=apim:api_publish" -H "Authorization: Basic eGU5NGIxcXJlTFZscGxIalpHd0x0ZnJtdjM4YTpmT3VhMmNmc2RUb3dOMGFwekNjSjJ1SWZMTVlh" https://127.0.0.1:8243/token

Response:

{"scope":"apim:api_publish","token_type":"Bearer","expires_in":3600,"refresh_token":"c99a3435230adceee296ec9517261b2a","access_token":"a44c384173298dc7db2b1a3a4fffc047"}

Publish the API:

curl -H "Authorization: Bearer a44c384173298dc7db2b1a3a4fffc047" -X POST "http://127.0.0.1:9763/api/am/publisher/v0.9/apis/change-lifecycle?apiId=96758d8d-b121-4ae9-a752-76d138295f3e&action=Publish" -v

Response:

< HTTP/1.1 200 OK
< Date: Sat, 26 Mar 2016 07:53:51 GMT
< Content-Length: 0
< Server: WSO2 Carbon Server

Accessing Store REST API

Like we did for accessing API Publisher, first of all we need to create an OAuth App using DCR endpoint to access Store REST API.

Creating an OAuth app to access Store REST API

curl -X POST -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "Content-Type: application/json" -d @payload.json http://localhost:9763/client-registration/v0.9/register

Response:

{"appOwner":null,"clientName":null,"callBackURL":"www.google.lk","isSaasApplication":true,"jsonString":"{\"username\":\"admin\",\"redirect_uris\":\"www.google.lk\",\"client_name\":\"admin_rest_api_store\",\"grant_types\":\"urn:ietf:params:oauth:grant-type:saml2-bearer iwa:ntlm implicit refresh_token client_credentials authorization_code password\"}","clientId":"asjOw7p7XpmtFoLRHHP2i290WT0a","clientSecret":"baTZ4fYKXb04Ab3xhLVBecBGdI4a"}

Now we have created an OAuth App with clientId=asjOw7p7XpmtFoLRHHP2i290WT0a and cliendSecret=baTZ4fYKXb04Ab3xhLVBecBGdI4a. We can use it to generate access tokens in subsequent calls.

curl -k -d "grant_type=password&username=admin&password=admin&scope=apim:subscribe" -H "Authorization: Basic YXNqT3c3cDdYcG10Rm9MUkhIUDJpMjkwV1QwYTpiYVRaNGZZS1hiMDRBYjN4aExWQmVjQkdkSTRh" https://127.0.0.1:8243/token

Response:

{"scope":"apim:subscribe","token_type":"Bearer","expires_in":3600,"refresh_token":"4c67b8741c1eb3d07613af15776fd056","access_token":"1a2863559e2a017924a3a3bba9708b8f"}

Now we have received an OAuth access token 1a2863559e2a017924a3a3bba9708b8f for apim:subscribe scope. In API Managet 1.10.0 REST API, you can use the above token to all store related operations

View APIs in Store

We use /apis API in Store REST API.

curl -H "Authorization: Bearer 1a2863559e2a017924a3a3bba9708b8f" http://127.0.0.1:9763/api/am/store/v0.9/apis -v

< HTTP/1.1 200 OK
< Date: Sat, 26 Mar 2016 08:13:13 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Server: WSO2 Carbon Server
<

{
"previous": "",
"list": [{
"provider": "admin",
"version": "1.0",
"name": "CalculatorSampleAPI",
"context": "/samplecalc/1.0",
"id": "96758d8d-b121-4ae9-a752-76d138295f3e",
"description": "Simple calculator API to perform addition, subtraction, multiplication and division.",
"status": "PUBLISHED"
}],
"count": 1,
"next": ""
}

Creating an Application

To create applications we need to use /applications API as with application details in body as a POST request.

curl -H "Authorization: Bearer 1a2863559e2a017924a3a3bba9708b8f" -H "Content-Type: application/json" -X POST -d @payload.json "http://127.0.0.1:9763/api/am/store/v0.9/applications" -v

payload.json:

{
"callbackUrl": "www.google.com",
"throttlingTier": "Unlimited",
"description": "sample app description",
"name": "sampleapp"
}

Response:

< HTTP/1.1 201 Created
< Location: http://localhost:9763/api/am/store/v0.9/applications/3fa24818-5c42-43fa-9b87-9fa2be06e3fe
< Date: Sat, 26 Mar 2016 08:11:09 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Server: WSO2 Carbon Server
<

{
"groupId": null,
"callbackUrl": "www.google.com",
"subscriber": "admin",
"throttlingTier": "Unlimited",
"applicationId": "3fa24818-5c42-43fa-9b87-9fa2be06e3fe",
"name": "sampleapp",
"keys": [],
"description": "sample app description",
"status": "APPROVED"
}

If you encounter the following error for first time accessing Store applications API after the user is added as a subsciber. Please try login to the Store web application once (or you can use old jaggery Store login API only once) and then invoke the applications API. This is a known limitation in API Manager 1.10.0.

< HTTP/1.1 500 Internal Server Error
< Date: Sun, 20 Mar 2016 13:47:21 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: close
< Server: WSO2 Carbon Server
<
* Closing connection 0
{"moreInfo":"","code":500,"description":"The server encountered an internal error. Please contact administrator.","message":"Internal server error","error":[]}

Generate application keys

To generate keys to the new application we need to use /generate-keys API. We need to provide application-id as a parameter and some other details as the payload.

curl -H "Authorization: Bearer 1a2863559e2a017924a3a3bba9708b8f" -H "Content-Type: application/json" -X POST -d @payload.json  "http://127.0.0.1:9763/api/am/store/v0.9/applications/generate-keys?applicationId=3fa24818-5c42-43fa-9b87-9fa2be06e3fe" -v

payload.json:

{
  "validityTime": "3600",
  "keyType": "PRODUCTION",
  "accessAllowDomains": ["ALL"
  ]
}

Response:

< HTTP/1.1 200 OK
< Date: Sat, 26 Mar 2016 08:17:03 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Server: WSO2 Carbon Server
<

"consumerKey": "GeJPyO1_sFHQ7sfyhwYd87TN_Kka",
"consumerSecret": "7QCdD4FgPCRvRnWnFuWPTmYos0Aa",
"keyState": "APPROVED",
"keyType": "PRODUCTION",
"supportedGrantTypes": ["urn:ietf:params:oauth:grant-type:saml2-bearer", "iwa:ntlm", "refresh_token", "client_credentials", "password"],
"token": {
"validityTime": 3600,
"accessToken": "0dc6c1e6b58be2410c779e1366aced4c",
"tokenScopes": ["am_application_scope", "default"]
}
}

Subscribe to the API

To add a subscription with the sample calculater API we added and the sample application we need to use /subscriptions API.

curl -H "Authorization: Bearer 1a2863559e2a017924a3a3bba9708b8f" -H "Content-Type: application/json" -X POST  -d @payload.json "http://127.0.0.1:9763/api/am/store/v0.9/subscriptions" -v

payload.json:

{
    "tier": "Unlimited",
    "apiIdentifier": "96758d8d-b121-4ae9-a752-76d138295f3e",
    "applicationId": "3fa24818-5c42-43fa-9b87-9fa2be06e3fe"
}

Response:

< HTTP/1.1 201 Created
< Location: http://localhost:9763/api/am/store/v0.9/subscriptions/558a42a6-8c8e-4bfa-9305-7a9912d0cabe
< Date: Sat, 26 Mar 2016 08:14:37 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Server: WSO2 Carbon Server
<

{
"tier": "Unlimited",
"subscriptionId": "558a42a6-8c8e-4bfa-9305-7a9912d0cabe",
"apiIdentifier": "admin-CalculatorSampleAPI-1.0",
"applicationId": "3fa24818-5c42-43fa-9b87-9fa2be06e3fe",
"status": "UNBLOCKED"
}

Invoke the API

Now we are done with Publisher and Store REST APIs for creating APIs, applications, generating keys and adding subscriptions. Lets spend a small amount of time trying invoking our newly created API to make sure everything is fine :).

We got the clientId and clientSecret as the response when creating our application "sampleapp". We can use that to generate an access token to invoke the API. We add base64Encoded as Authorization : Basic header like we did before.

curl -k -d "grant_type=password&username=admin&password=admin" -H "Authorization: Basic R2VKUHlPMV9zRkhRN3NmeWh3WWQ4N1ROX0trYTo3UUNkRDRGZ1BDUnZSblduRnVXUFRtWW9zMEFh" http://127.0.0.1:8280/token -v

< HTTP/1.1 200 OK
< Content-Type: application/json
< Pragma: no-cache
< Cache-Control: no-store
< Date: Sat, 26 Mar 2016 08:21:14 GMT
< Transfer-Encoding: chunked
<

{"scope":"default","token_type":"Bearer","expires_in":79,"refresh_token":"1ef74fa5bb4d71d0d5fa5956a44dfb98","access_token":"c4eecb842f00de8ed5aedf66e1b72bb5"}

Now we invoke our sample API using that access token.

curl -X GET --header "Accept: application/json" --header "Authorization: Bearer c4eecb842f00de8ed5aedf66e1b72bb5" "https://127.0.0.1:8243/samplecalc/1.0/add?x=1&y=2" -k -v

< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: authorization,Access-Control-Allow-Origin,Content-Type
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET
< Content-Type: application/json
< Date: Sat, 26 Mar 2016 09:16:10 GMT
< Transfer-Encoding: chunked
<
{"answer": "3.0"}

API Manager New REST API - The First Step

There are three Jax-RS web applications related to API Manager new REST API.

1. Publisher Jax-RS web application
The JaxRS REST API web application which is responsible for all Publisher web app related functions.

2. Store Jax-RS web application
The JaxRS REST API web application which is responsible for all Store web app related functions.

3. Client-Registration Jax-RS web application

The JaxRS REST API web application which is responsible for creating OAuth applications in order to access Store or Publisher REST APIs.

Invoke DCR and create an OAuth Application

For this step you can also refer API Manager REST API documentation [1], but I will state the basic steps in brief.

As the first step you have to invoke the client-registration API. This is a new JaxRS web app, introduced in API Manager 1.10.0 version to create OAuth applications to invoke the new REST API. It has initial version of v0.9.

The API is secured with Basic authentication so you need to provide base64encoded(username:password) as the Authorization: Basic header.

curl -X POST -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "Content-Type: application/json" 
-d @payload.json http://localhost:9763/client-registration/v0.9/register

Sample payload.json file:

{
    "callbackUrl": "www.google.lk",
    "clientName": "rest_api_publisher",
    "tokenScope": "Production",
    "owner": "admin",
    "grantType": "password refresh_token",
    "saasApp": true
}

In this payload you specify "clientName" which is the OAuth application name.

After sending the above request you will get a response like this:

HTTP/1.1 201 Created
Date: Sun, 20 Mar 2016 03:29:57 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: WSO2 Carbon Server

{
   "jsonString":"{\"username\":\"admin\",\"redirect_uris\":\"www.google.lk\",\"client_name\": \"admin_rest_api_publisher\",\"grant_types\":\"urn:ietf:params:oauth:grant-type:saml2-bearer iwa:ntlm implicit refresh_token client_credentials authorization_code password\"}",
   "appOwner":null,
   "clientName":null,
   "callBackURL":"www.google.lk",
   "isSaasApplication":true,
   "clientId":"xe94b1qreLVlplHjZGwLtfrmv38a",
   "clientSecret":"fOua2cfsdTowN0apzCcJ2uIfLMYa"
}

Now you have successfully created an OAuth application. Now you can invoke the Token API [2] to create an access token using the received client ID and client Secret. This is exactly same as when you invoke token API to create an access token to invoke an API published in API Manager.

This supports all OAuth2 grant types that API Manager supports and here I am using password grant type to create an access token.

curl -k -d "grant_type=password&username=admin&password=admin&scope=<scopes>" 
-H "Authorization: Basic <base64encoded(clientid:clientsecret)> https://127.0.0.1:8243/token

Scopes

One thing to note here is the scope field. In new REST API there is a specific scope mapped to resource verb combinations. For example if you need to invoke Get All APIs in Publisher REST API, your token need to have apim:api_view scope.

The REST API consists of following OAuth scopes. As an API creator, publisher or subscriber you need to have access to following set of scopes:

As an API Creator:

• apim:api_view  - to view APIs
• apim:api_create - to create new APIs
• apim:subscription_view - to view subscriptions of APIs
• apim:subscription_block - to block/unblock subscription of APIs
• apim:tier_view - to view tiers

As an API Publisher:

• apim:api_view  - to view APIs
• apim:api_publish - to change lifecycle states of APIs (publish, deprecate, retire APIs etc)

As an API Subscriber:

• apim:api_subscribe - create applications, view applications, subscribe to APIs and all Store related operations

As an Administrator:

• ++ all creator scopes
• ++ all publisher scopes
• ++ all subscriber scopes
• apim:tier_manage - to block unblock subscribing to tiers in Store based on user's roles

Tip
--------
You can just try asking for an invalid random scope and check what you receive.

curl -k -d "grant_type=password&username=admin&password=admin&scope=random_invalid_scope" 
-H "Authorization: Basic eGU5NGIxcXJlTFZscGxIalpHd0x0ZnJtdjM4YTpmT3VhMmNmc2RUb3dOMGFwekNjSjJ1SWZMTVlh" 
https://127.0.0.1:8243/token -v

Response:

< HTTP/1.1 200 OK
< Content-Type: application/json
< Pragma: no-cache
< Cache-Control: no-store
< Date: Sun, 20 Mar 2016 07:14:59 GMT
< Transfer-Encoding: chunked
<

* Connection #0 to host 127.0.0.1 left intact
{"scope":"default","token_type":"Bearer","expires_in":3600,"refresh_token":"1e2f7d03297f51b2bf4b0762efa9546a",
"access_token":"081f892624caeb3be2b1a1ab22b00316"}

You can see you receives an scope called default but not what you requested.
--------

You can request single or many scopes at once. When requesting multiple scopes you need to give them space separated.

Let's request all scopes related to an API creator. See the sample curl command below.

curl -k -d "grant_type=password&username=admin&password=admin&scope=apim:api_view apim:api_create 
apim:subscription_view apim:subscription_block apim:tier_view" 
-H "Authorization: Basic eGU5NGIxcXJlTFZscGxIalpHd0x0ZnJtdjM4YTpmT3VhMmNmc2RUb3dOMGFwekNjSjJ1SWZMTVlh" 
https://127.0.0.1:8243/token -v

You will receive following response:

< HTTP/1.1 200 OK
< Content-Type: application/json
< Pragma: no-cache
< Cache-Control: no-store
< Date: Sun, 20 Mar 2016 07:09:41 GMT
< Transfer-Encoding: chunked
<
* Connection #0 to host 127.0.0.1 left intact
{"scope":"apim:api_create apim:api_view apim:subscription_block apim:subscription_view apim:tier_view",
"token_type":"Bearer","expires_in":3600,"refresh_token":"dc760c94e3c8a5c4c3d263b14708b275","access_token":"fb3db536b07f2a1da0185d82b0887848"}

You can see you received all scopes you requested -  "scope":"apim:api_create apim:api_view apim:subscription_block apim:subscription_view apim:tier_view" .  

Before going further you have to make sure you have received the scopes you requested.

Invoking a sample API

Now we are done with getting an access token. Lets try to invoke an API in Publisher.

An easiest API for a beginner is Get All Tiers API as it does not require any prerequisites. See that API in documentation [3]

Request path                  : https://localhost:9443/api/am/publisher/v0.9/apis/tiers/api
Request HTTP method  : GET
Scope                             : apim:tier_view

Use the following curl command. Replace the access token with what you received from token API. Lets use the access token we got from invoking the token API. We have to use it as Authorization: Bearer header. Just same as when we invoke an API published in API Manager.

curl -k -H "Authorization: Bearer fb3db536b07f2a1da0185d82b0887848"
https://127.0.0.1:9443/api/am/publisher/v0.9/tiers/api -v

You will be able to see following response.

< HTTP/1.1 200 OK
< Date: Sun, 20 Mar 2016 08:19:56 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Server: WSO2 Carbon Server
<
* Connection #0 to host 127.0.0.1 left intact
{  
   "previous":"",
   "list":[  
      {  
         "unitTime":60000,
         "tierPlan":"FREE",
         "tierLevel":"api",
         "stopOnQuotaReach":true,
         "requestCount":1,
         "description":"Allows 1 request(s) per minute.",
         "name":"Bronze",
         "attributes":{  

         }
      },
      {  
         "unitTime":60000,
         "tierPlan":"FREE",
         "tierLevel":"api",
         "stopOnQuotaReach":true,
         "requestCount":20,
         "description":"Allows 20 request(s) per minute.",
         "name":"Gold",
         "attributes":{  

         }
      },
      {  
         "unitTime":60000,
         "tierPlan":"FREE",
         "tierLevel":"api",
         "stopOnQuotaReach":true,
         "requestCount":5,
         "description":"Allows 5 request(s) per minute.",
         "name":"Silver",
         "attributes":{  

         }
      },
      {  
         "unitTime":0,
         "tierPlan":null,
         "tierLevel":"api",
         "stopOnQuotaReach":true,
         "requestCount":0,
         "description":"Allows unlimited requests",
         "name":"Unlimited",
         "attributes":{  

         }
      }
   ],
   "count":4,
   "next":""
}

References:

[1] https://docs.wso2.com/display/AM1100/apidocs/publisher/index.html#guide
[2] https://docs.wso2.com/display/AM1100/Token+API
[3] https://docs.wso2.com/display/AM1100/apidocs/publisher/index.html#!/operations#TiersApi#tiersTierLevelGet

How to create a custom URL for a tenant for API Manager Store

Default URL for API Manager Store is https://<hostname>:9443/store. Instead of that you can use a URL like https://store.wso2apps.com to access the Store.

Setting up a custom URL for Store:

 To enable the custom URL using a reverse proxy you need to install nginx. To install nginx and setup SSL certificates you can follow.

http://sanjeewamalalgoda.blogspot.com/2014/12/configure-wso2-api-manager-with-reverse.html

Installing Nginx:
sudo apt-get install nginx

Creating a SSL certificate:
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt

Go to <APIM_HOME>/repository/resources/security folder using terminal.

Then use following command to add the certificate in to client trust store.

keytool -import -file /etc/nginx/ssl/nginx.crt1 -keystore client-truststore.jks -storepass wso2carbon -alias wso2carbon2

Use the following command to open nginx configuration in gedit.

sudo gedit /etc/nginx/sites-enabled/default

Add the following configuration:

You can use a host name you like such as "store.api.com" which I have used here.



You need to have put it in your /etc/hosts. For example you if you need to name your host as "store.apim.com" add  the following entry to /etc/hosts file. Replace the IP with your computer's IP.

10.100.7.77 store.apim.com

To start nginx use the following command.

sudo /etc/init.d/nginx start

After that start the server and go to "https://store.apim.com/" url from the browser and you can see it will directly go to the tenant store.

Few things to take a note:

1. proxy_set_header X-WSO2-Tenant "ten5.com";
• This a WSO2 specific header we set to specify the tenant the custom URL should apply. Not like the normal store (which will show all the tenants when go to the store) it will directly go to the specified tenant in the header.
2. proxy_redirect  https://store.apim.com/store/ /; and proxy_redirect  https://localhost:9443/store/ /;
• This need to be set to re-write the location header to the correct URL when the server send HTTP re-directions to the browser.

Configuring Store web app

Add following setting to site.conf of store webapp.

    "reverseProxy" : {
        "enabled" : "auto",    // values true , false , "auto" - will look for  X-Forwarded-* headers
        "host" : "sample.proxydomain.com", // If reverse proxy do not have a domain name use IP
        "context":"",
        "tenantHeader" : "X-WSO2-Tenant"
    }

Setup custom domain mapping in registry

You can change the gateway URL appears in store swagger as per your own domain. What you need to do is, configuring custom domain mapping for gateway in super tenant registry following manner.

Login to carbon management console as admin@carbon.super

Create a new resource in following path: (Resource name is the tenant domain name. In my example, ten1.com)

_system/governance/customurl/api-cloud/ten1.com/urlMapping/ten1.com

Add the following content:

{
    "tenantDomain": "ten1.com",
    "store" : {
        "customUrl" : "store.apim.com"  // custom domain for store
    },
    "gateway" : {
        "customUrl" : "gw.apim.com"   // custom domain for gateway
    }
}


This is how this would look like in management console.

Now you can see the custom URLs appears in swagger console instead of the default gateway URL.

Once you create an API and publishing it from tenant's Publisher, you can see following dialog appears. If you click on Goto API Store, you will also observe that you will go to your custom store domain; not the default store URL. This is actually redirecting to what you have configured in "customDomain" before.

Writing a custom NTLM grant handler and a sample client for API Manager with handshake support

NTLM is a challange/response based authentication protocol which is proprietary for Microsoft. If you are new to NTLM and need to have a basic idea what is happening you can read my previous blog post [1].

This blog explains how to write a custom grant handler to WSO2 API Manager to support NTLM grant type.

If you are going to try this please note you should be in a Windows environment. If you are trying this on a single machine (Client and server user accounts on a same computer), you would be able to try this without any issues (I have tried on a Windows 7/8 environments). But if you need to try an actual production environment (clients and server are different computers) , first of all you need to setup a Active Directory Domain. You can follow link [2] to setup that.

In this process we need to deal with Windows Platform related APIs. For that we can use Java Native Access (JNA) [3] library which is a simple way of native access without requiring to write JNI (Java Native Interface) code. There is another library which is built on top of JNA which is Waffle [4] which encapsulates all functionality you need to implement user authentication.

Following Maven dependencies can be used for setting up JNA and Waffle respectively.



First step of the NTLM handshake is to create the Type1 token by the client.

Generated token is then encoded into Base64 and should be send as a header.

Server (APIM) recieves the token from the request header. Then it validates the Type 1 token and generates a Type 2 token. It is sent back to the client. Then client should generate a Type 3 token based on Type 2 token. It is sent to the server as a Header.
Once server receives a token, it first determines the type of it. Type is the value of the 8th byte of the Base64 decoded NTLM token. If the server determines it as a Type 1 token, then it validates it and generates a Type 2 token based on that.
Following function can be used to determine the type of the token.

If the server determines it as a Type 1 token, then it validates it and generates a Type 2 token based on that. If it is Type 3 token, it will validate it and it will then identifies the user details communicating with the domain controller.
To write a custom grant handler we can basically follow the doc [5]. That includes a sample [6] code too. When writing our handler we need to basically extend AbstractAuthorizationGrantHandler when writing our class.
Following is the grant handler code.

The link [7] consists of full code of client and server.

Once you completed writing the handler code, you can do the following configuration to enable the new custom grant type.

There is a section called "SupportedGrantTypes" in identity.xml in /repository/conf folder. (If you are using APIM 1.10.x this can be found at /repository/conf/identity folder). Add your new grant type to that section as a new SupportedGrantType. Correctly put the fully qualified class name of the custom grant handler under . 



After setting this you can start the server. You can use the provided sample client at [7] to test the new NTLM grant handler. You can follow the Readme.txt file which is added there.


[1] NTLM Authentication Basics : http://malinthaprasan.blogspot.com/2015/10/ntlm-authentication-basics.html
[2] Setting up an Active Directory Domain: http://roshanwijesena.blogspot.com/2015/05/ntlm-grant-type-support-with-wso2-api.html
[3] Java Native Access (JNA) Library: https://github.com/java-native-access/jna
[4] Waffle : https://github.com/dblock/waffle
[5] Writing a custom grant type : https://docs.wso2.com/display/IS500/Writing+a+Custom+OAuth+2.0+Grant+Type
[6] Custom grant type sample code : https://svn.wso2.org/repos/wso2/people/asela/oauth/custom-grant/
[7] Sample NTLM grant handler with handshake support and a sample client code: https://drive.google.com/open?id=0B1jOwGWdNiSNMEw3UFJ1R3Z4WFU

NTLM Authentication basics

Introduction

NTLM which is denoted as NT LAN Manager (NTLM) is a suite of Microsoft security protocols (This is a Proprietary Microsoft authentication protocol) that provides authentication, integrity, and confidentiality to users which is used in various Microsoft network protocol implementations.

Basic Structure

This typically involves three systems as follows.

Three systems involved in NTLM authentication

Key Features 

Following are key features of the NTLM authentication scheme.

1. Uses a challenge-response mechanism for authentication, in which clients are able to prove their identities without sending a password to the server. 
2. Provides authentication, confidentiality and integrity
3. Enables Single Sign On.

The Three-Step Handshake

NTLM It consists of three messages, commonly referred to as Type 1 (negotiation), Type 2 (challenge) and Type 3 (authentication).

Following is the message flow that happens between the client and the server.

Three step handshake of NTLM authentication protocol

• (1) Client try to access the server's resource without authorization headers.

• (2) Server responds with 401 Unauthorized. With WWW-Authenticate: NTLM header, server requests the client to send NTLM authentication token.

• (3) Client re-sends the request with NTLM type 1 token. It initiates the NTLM authentication. This contains few information including the hostname and the domain name of the client [1].

NTLM Type 1 token structure

• (4) Server receives the Type 1 token from the client. In response to that, sever sends Type 2 token which consists of server's NTLM challenge. [2]

NTLM Type 2 token structure

• (5) In response to Type 2 token, client sends the Type 3 token as the final step in authentication. This demonstrates that the client has knowledge of the account password without sending the password directly. What actually client does is that it encrypts the server's challenge with its hashed password when generating the Type 3 token. Other than the response to the challenge, it consists the username too [3].

NTLM Type 3 token structure

 Once server receives the type 3 token, it sends the following three items to the domain controller [4].

• User name
• Challenge sent to the client
• Response received from the client

The domain controller uses the user name to retrieve the hash of the user's password from the Security Account Manager database. It uses this password hash to encrypt the challenge. The domain controller compares the encrypted challenge it computed to the response computed by the client. If they are identical, authentication is successful.

Weaknesses

1. Generated hashes of the client's password is not salted. So it remains same until the password is changed. So this is vulnerable to pass the hash [5] attack where once an attacker finds the hash once that can be used to encrypt the server's challenge successfully. 
2. NTLM does not support mutual authentication. i.e only client is authenticated. The client has to assume that the server is legitimate.
3. The password hash in NTLM is exposed each time the client uses NTLM for authenticating to a server. There are tools exists that scan network traffic for NTLM password hashes, capture them and then do a brute-force crack on them to derive the user's password.

Because of this problems Microsoft has recommended Kerberos protocol over NTLM which gives a greater security. It provides mutual authentication. The password hash is less frequently exposed as it is only required to send when the user requests a TGT (Ticket Granting Ticket) [6]. But still there are situations that Kerberos protocol cannot be applied. In such occasions NTLM is still used.

References

[1] NTLM Type 1 message Example : http://davenport.sourceforge.net/ntlm.html#type1MessageExample
[2] NTLM Type 2 message Example :
http://davenport.sourceforge.net/ntlm.html#type2MessageExample
[3] NTLM Type 3 message Example :
http://davenport.sourceforge.net/ntlm.html#type3MessageExample
[4] Microsoft NTLM : https://msdn.microsoft.com/en-us/library/windows/desktop/aa378749(v=vs.85).aspx
[5] Pass the hash : https://en.wikipedia.org/wiki/Pass_the_hash
[6] Authentication in Windows Server 2008 R2 and Windows 7 : http://windowsitpro.com/security/authentication-windows-server-2008-r2-and-windows-7

Introducing Default API Version for WSO2 API Manager - My Internship at WSO2

I worked 6 months at WSO2 as an Software Engineering intern from November 2013. That period of 6 months is undoubtedly the best ever period of time I had spend as a university student so far. There were 24 students from our batch with me in training. Other than the great experience working with an open source community, we had a great time filled with entertainment.

After we finished the intern project(The first phase of the training), I and two of interns from UoM were assigned to WSO2 API Manager Team. I had a great time with them filled with team work and got a lot of experience. It was when I had a chance to develop this new feature to WSO2 API Manager Product, "Introducing Default API Version".

Introducing a Default Version to API 

One of the main challenges that API Manager faces is API Versioning. Previously when invoking an API in WSO2 API Manager, the API version field was mandatory. With this feature, the provider can select one of the APIs with the same name as Default and the subscribers who wish to take the benefits of the Default API can subscribe to it using the API Store . When a subscriber invoke an API the usual curl command would look like this:

curl -H "Authorization :Bearer <token>" http://<host>:<port>/<apiname>/<apiversion>/<resource>

By introducing this feature a subscriber can invoke the same API like this without the version field.
curl -H "Authorization :Bearer <token>" http://<host>:<port>/<apiname>/<resource>

By this feature, API provider is given the facility to select any of the APIs any time as Default Version.

Benefits for the API Providers and Subscribers with this feature:

APIs change time to time like everything. May be there are new features introduced, bugs are resolved, improved the performance and etc. Then usually what happens is, the provider need to introduce another API version to the API Store. Then the subscribers of same API see that in the API store and need to be resubscribed. By introducing this feature, the provider can easily avoid that by selecting the Latest Stable version as the Default API. When there is such an occasion to introduce a new API, the provider can select it as the Default version. Then the current subscribers easily are switched to the new API without re-subscription. This avoids the delay occurring with re-subscription period.

With this feature, the subscribers of the Default API Version can easily be switched to another API with a very small effort of the provider side.

Implementation

High level implementation can be summarized by this diagram.

By this new implementation, a new API proxy is introduced to the Synapse Gateway. When defining an API to the gateway usually the version is specified but for this API it is not. It basically does the relevant routing the configured API as default when it receives a request.

We further discussed about the architecture of the solution and the following was considered.

• Introduce a new attribute to the API artifact to indicate whether an API is Default or not

In API Manager, registry is used as the storage for the meta data and resources. Accordingly the currently created APIs' meta data are stored there. For the new feature a new field is introduced for the api.rxt template to hold the default API version attribute.

• Creating a separate table in the API Manager database to keep the information of APIs which are selected as default.

Above was a sample h2 database table which is used to store Default API related information. There are two default API related columns which are DEFAULT_API_VERSION and PUBLISHED_DEFAULT_API_VERSION. According to the API life-cycle the default version API can also be in those states. So DEFAULT_API_VERSION field is used to hold the actual version that the provider selected as default and the PUBLISHED_DEFAULT_API_VERSION is used to hold the API version that is in published state. According to that the default API can be in following states.

There were two reasons behind maintaining two such fields. In this feature, the provider can change default API version anytime(Default version can be switched to another version any time. But in that occasions we have to make sure that the existing users do not break after changing the Default API. So once the subscribers are using one API as Default there is a possibility that the provider select an API as default which is in CREATED state. So now all the Apps which are using the default API will break. To avoid that a separate column is maintained to store the published version of the Default API. In such occasions the apps are temporary routed to the previous default version until the current Default API is taken to the PUBLISHED state.

The other reason is that currently the apps are authenticated at the gateway when they are invoking an API. For that, a token which is generated through the Thrift server is used. At the gateway, there is a handler called Authentication Handler and it does that validation. The problem which arise when changing the Default Version API is that the users are now routed to a new API and the users may or may not be subscribed to that API. So if the normal authentication flow applied, most probably the validation will fail and the app will not be allowed to call the new API. So a slight change is done to the key validation process such a way that if a user is previously subscribed to the Default API and now accessing the Published version of the Default API(which is stored in PUBLISHED_DEFAULT_API_VERSION field) then he is allowed to invoke the Default API. Then it make sure that the existing user does not break after changing the Default API.

The following State diagram briefs the above states of the Default API.

When it is needed to add an API to the gateway an API proxy should be defined in the gateway. Currently Synapse gateway is used(Actually Apache Synapse is the core of the WSO2 ESB which is the gateway of API Manager). The API proxy is defined using XML.

A sample API proxy which is published in the gateway uses the following pattern in XML.

<api name="provider1--youtube" context="/youtube">   
<resource methods="POST GET DELETE OPTIONS PUT" url-mapping="/*">
     <inSequence>
              <!-- Processed when a request received from the client -->    
              <send>
                 <endpoint>
                    <!-- defines the back end endpoints for the API -->   
                    <http uri-template="http://gdata.y">
                       <timeout>
                          <duration>30000</duration>
                          <responseAction>fault</responseAction>
                       </timeout>
                           …..
                           …..
                    </http>
                 </endpoint>
              </send>
     </inSequence>
     <outSequence>
         <!-- Processed when a response received from the backend -->
        <send/>
     </outSequence>
  </resource>
  <handlers>
     <handler>
     <!-- defines handles for the API to record statistics etc..-->
     <handler class="org.wso2.carbon.apimgt.gateway.handlers.common.SynapsePropertiesHandler"/>
     </handler>
  </handlers>
</api>

The same sample is followed when defining a Default API proxy configuration to the gateway. This is the sample API that is used to define a Default API in the gateway.

<api name="provider1--youtube" context="/youtube">   
<resource methods="POST GET DELETE OPTIONS PUT" url-mapping="/*">
     <inSequence>
 
<property name="isDefault" expression="get-property('transport', 'WSO2_AM_API_DEFAULT_VERSION')"/>
        <filter source="get-property('isDefault')" regex="true">
           <then>
              <log level="custom">
                 <property name="STATUS" value="Faulty invoking through default API.Dropping message to avoid recursion.."/>
              </log>
              <payloadFactory media-type="xml">
                 <format>
                    <am:fault xmlns:am="http://wso2.org/apimanager">
                       <am:code>500</am:code>
                       <am:description>Faulty invoking through default API</am:description>
                    </am:fault>
                 </format>
                 <args/>
              </payloadFactory>
              <property name="HTTP_SC" value="500" scope="axis2"/>
              <property name="RESPONSE" value="true"/>
              <send/>
           </then>
           <else>
              <header name="WSO2_AM_API_DEFAULT_VERSION" scope="transport" value="true"/>
              <property name="uri.var.portnum" expression="get-property('http.nio.port')"/>  
              <send>
                 <endpoint>
                    <http uri-template="http://localhost:{uri.var.portnum}/youtube/2.0">
<timeout>
                          <duration>30000</duration>
                          <responseAction>fault</responseAction>
                       </timeout>
                    </http>
                 </endpoint>
              </send>
           </else>
        </filter>
     </inSequence>
     <outSequence>
        <send/>
     </outSequence>
  </resource>
  <handlers>
     <handler class = "org.wso2.carbon.apimgt.gateway.handlers.common.SynapsePropertiesHandler"/>
  </handlers>
</api>

First part of the definition is done as a fail-safe operation. I had fixed a bug related to synapse APIs. But if anything went wrong or the patch is not properly applied there was a big possibility to message received to the Default API is being blocked and recursively processed. So to avoid that before sending the message out, a header called WSO2_AM_API_DEFAULT_VERSION is set. Then if the same message circulated to the Default API again, the header will be there and it can be easily identified that the message has been circulated. So if a message is received, it will check for the above header and if it is there, the message will be dropped with a server log and sending a fault message to the client.

I used a separate handler to the above API which is SynapsePropertiesHandler. When forwarding to the relavent API the Default API should know which port the synapse is up. Hard coding it can cause problems in the future if the port of the server was changed after creating several number of APIs, because the port mentioned in the proxy configuration is not changed. So the messages will not be sent to the API. The best way to do is taking that value from the synapse message context. The handler fetches that value on runtime and sets to a property called {http.nio.port}. That value is taken as the port when forwarding the messages.

Following is the implementation of that simple handler class. When whiting a handler the class should be extended from the synapse AbstractHandler class.

public class SynapsePropertiesHandler extends AbstractHandler{

   public boolean handleRequest(MessageContext messageContext) {
       String httpport = System.getProperty("http.nio.port");
       messageContext.setProperty("http.nio.port", httpport);
       return true;
   }

   public boolean handleResponse(MessageContext messageContext) {
       return true;
   }
}

Fields of the above API proxy can be changed according to the API which is being set as Default. Those parameters are API name, provider name, version to be routed the message. So when producing the above proxy definition certain fields should be taken care as variables.

When producing existing API configurations API Manager uses Apache Velocity. Apache Velocity is a Java-based template engine. It primarily permits web page designers to reference methods defined in Java code. Other than developing web pages Velocity can be used to generate SQL, PostScript and other output from templates. It can be used either as a standalone utility for generating source code and reports, or as an integrated component of other systems. 

In Velocity VTL(Velocity Template Language) is used. Inside the template we can define variables so that they can be set as Java objects during processing the template. Following is the brief of the template in VTL I used as the Default API.

<api xmlns="http://ws.apache.org/ns/synapse"  name="$!apiName" context="$! apiContext">
       <resource methods="POST GET DELETE OPTIONS PUT" url-mapping="/*">
           <inSequence>
<property name="isDefault" expression="get-property('transport', 'WSO2_AM_API_DEFAULT_VERSION')"/>
               <filter source="get-property('isDefault')" regex="true">
                   <then>
                       ...
                       <send/>
                   </then>
                   <else>
                       ...
                       <send>
                           <endpoint>
                               <http uri-template="http://localhost: {uri.var.portnum}/$!{fwdApiContext}/$!{defaultVersion}">
                                   ...
                               </http>
                           </endpoint>
                       </send>
                   </else>
               </filter>
           </inSequence>
           <outSequence>
               <send/>
           </outSequence>
       </resource>
       <handlers>
           <handler …/>
       </handlers>
</api>

literals starting from $! are variables that are defined in VTL.  So references are set to that variables during the runtime like this.

public String getConfigStringForDefaultAPITemplate(String defaultVersion) throws APITemplateException {
       StringWriter writer = new StringWriter();
       try {
           VelocityEngine velocityengine = new VelocityEngine();
           velocityengine.init();

           ConfigContext configcontext= new APIConfigContext(this.api);

           VelocityContext context = configcontext.getContext();
           context.put("defaultVersion",defaultVersion);
           String fwdApiContext=this.api.getContext();
           if(fwdApiContext!=null && fwdApiContext.charAt(0) == '/')
               fwdApiContext=fwdApiContext.substring(1);
           context.put("fwdApiContext",fwdApiContext);

           Template t = velocityengine.getTemplate(this.getDefaultAPITemplatePath());

           t.merge(context, writer);
       } catch (Exception e) {
           log.error("Velocity Error", e);
           throw new APITemplateException("Velocity Error", e);
       }
       return writer.toString();
   }

VelocityContext instance is used to share the references to build the outcome using the template. This implementation uses a HashMap (java.util.HashMap ) for data storage. To generate the outcome Context and the Template is merged.

After the required configurations were generated they should be published in the gateway. The API Manager gateway such that several gateways can exist. In such occasions the configuration should be copied to all the other instances as well.