mscharhag, Programming and Stuff;

A blog about programming and software development topics, mostly focused on Java technologies including Java EE, Spring and Grails.

  • Tuesday, 21 July, 2020

    Validating and documenting JSON with JSON Schema

    JSON Schema is a way to describe a JSON document. You can think of XML Schema for JSON. It allows you to define required elements, provide validation constraints and documentation.

    I think the easiest way to explain JSON Schema is to look at an example snippet of JSON and the corresponding JSON Schema. So, I created an image that shows both and is hopefully self explanatory:

    json-schema

    (Click to enlarge, if you are on a mobile device)

    A JSON Schema validator can be used to validate a JSON document against a JSON Schema. This can be done online (e.g. https://www.jsonschemavalidator.net/) or using a library of your favourite programming language. In the implementations section of json-schema.org you can find a couple of libraries to work with JSON Schema.

    In case you want to copy/paste some of the sections of the image: You can find the example as text on GitHub.

  • Wednesday, 15 July, 2020

    Common HTTP Status codes

    What is a HTTP status code?

    HTTP status codes are part of the status-line of a HTTP response. These 3-digit integer codes indicate the result of the servers attempt to satisfy the request.

    The first digit of the status-code is used to categorize the response:

    • 1xx: Informal
    • 2xx: Success, the request has been understood and accepted
    • 3xx: Redirection, further action needs to be taken
    • 4xx: Client error, there was a problem with the request
    • 5xx: Server error, the request has been accepted, but the processing failed due to a server error

    Commonly used HTTP status codes

    Here is a list of status codes commonly used in web applications and REST APIs.

    HTTP 200 OK

    The request has succeeded. HTTP 200 is often used as the default status code to indicate that everything worked as expected. However, other 2xx status code can be more specific in certain situations.

    HTTP 201 Created

    The request has succeeded and a new resource has been created. HTTP 201 is often used after a resource has been created by a POST or PUT request (see REST: Creating resources). The Location header field can be used to return the URI(s) of the newly created resource. Note that the resource must be created before 201 is returned. If the resource is created later by an asynchronous process, the server should respond with HTTP 202 (Accepted, see below).

    HTTP 202 Accepted

    The request has been accepted, but the processing has not been completed. For example, the server might have saved the request, but the processing will be done later by a batch job. It is also possible that the request might be disallowed when processing actually takes place. In this case, the request has no effect. The response should include some indication about the processing status. It can also be a good idea to return a reference to a resource where the client can get the current processing status and the result once processing has been finished.

    HTTP 204 No Content

    The request succeeded, but there is no content to send for this request. When sending 204 the response body must be empty. Updated meta-information can be passed in response headers.

    HTTP 304 Not Modified

    This status code is used for caching purposes when a client issued a conditional GET request. Such a request has to contain a If-None-Match or If-Modified-Since header. 304 is returned with an empty response body to indicate that the resource has not been modified. In case the resource has been modified, the resource should be returned with a status code 200 (OK).

    HTTP 307 Temporary Redirect

    The URI of the target resource has been temporary changed and current URI is given in the response Location header. Temporary Redirect indicates that the client should use the original request URI for future requests.

    HTTP 308 Permanent Redirect

    Like 307 this status code is used when the target resource URI has been changed. The new URI is again given in the Location header. However, 308 indicates that the URI change is permanent and clients should use the updated URI for future requests.

    HTTP 400 Bad Request

    The request has been received, but the server is unable to process it due to malformed request syntax. The client should not repeat the request without modification. This status code is often returned if server side validation of the request data fails.

    HTTP 401 Unauthorized

    The request lacks credentials and cannot be authenticated. Note that this status code is badly named, it is used to indicate missing authentication not missing authorization. For missing authorization HTPP 403 is used. This status code is typically returned if the request does not include required authentication information such as passwords or tokens.

    HTTP 403 Forbidden

    The client is authenticated, but does not have the permission to use the given request method on the requested resource. It is also viable to respond with HTTP 404 (Not Found) in these situations if the server wants to hide the resource. The client should not repeat the request (without changing credentials). If the client is not authenticated at all, HTTP 401 should be returned instead.

    HTTP 404 Not Found

    The server has not found the requested URI and is therefore unable to process the request.

    HTTP 405 Method Not Allowed

    The server does not allow the requested HTTP method for the given URI. For example, the request specifies the PUT method for a resource that only supports GET and POST. The response must include an Allow header containing a list of valid request methods for the requested resource.

    HTTP 409 Conflict

    The request could not be completed due to a conflict with the current state of the resource. This code must only be used in situations where the user might be able to resolve the conflict and reissue the request. The response body should contain information about the conflict so the client is able to solve it. An example can be a resource update using PUT. Maybe the resource has been updated by another third-party request and the current request does not reflect the current resource state.

    HTTP 500 Internal Server Error

    The server has encountered a situation it is unable to handle. This is the standard status code for unexpected error on the server during the request processing.

     

  • Monday, 13 July, 2020

    Quick tip: ISO 8601 durations in Java

    Many developers know about the interchange formats for dates and times defined by ISO 8601. (For example 2007-08-31T16:47+00:00 which represents 16:47 on August 31, 2007 in UTC)

    However, what is not so well-known (at least in my experience), is that this standard also defines a format for durations.

    Here are a few examples:

    • P1Y - 1 year
    • P2M4D - 2 months and 4 days
    • P3Y6M4DT12H30M5S - 3 years, 7 months, 4 days, 12 hours, 30 minutes, and 5 seconds

    In Java it is quite easy to work with this format because the java.time API can automatically parse it to Duration and Period objects.

    Duration is used to work with hours and smaller time units while Period works with dates.

    For example:

    Duration duration = Duration.parse("PT12H"); // 12 hours
    Period period = Period.parse("P3Y6M"); // 3 years and 6 months

    We can now work with these instances, for example we can add period to a LocalDate:

    LocalDate result = LocalDate.of(2020, 8, 1).plus(period); // 2024-02-01
  • Wednesday, 8 July, 2020

    Getting started with Ktor

    Ktor (pronounced kay-tor) is an open source Kotlin framework for building asynchronous web applications. This post shows how to create a small RESTful CRUD service with Ktor.

    Getting started

    In this example we use Maven as build tool. Besides standard Kotlin dependencies we need to add the Ktor dependencies to our pom.xml:

    <project>
    
        <properties>
    		...
            <ktor.version>1.3.2</ktor.version>
        </properties>
    
        <repositories>
            <repository>
                <id>jcenter</id>
                <url>https://jcenter.bintray.com</url>
            </repository>
        </repositories>
    
        <dependencies>
            <dependency>
                <groupId>io.ktor</groupId>
                <artifactId>ktor-server-core</artifactId>
                <version>${ktor.version}</version>
            </dependency>
            <dependency>
                <groupId>io.ktor</groupId>
                <artifactId>ktor-server-netty</artifactId>
                <version>${ktor.version}</version>
            </dependency>
            <dependency>
                <groupId>io.ktor</groupId>
                <artifactId>ktor-gson</artifactId>
                <version>${ktor.version}</version>
            </dependency>
            <dependency>
                <groupId>ch.qos.logback</groupId>
                <artifactId>logback-classic</artifactId>
                <version>1.2.3</version>
            </dependency>
        </dependencies>
    	...
    </project>

    You can find the full pom.xml (including standard Kotlin dependencies and plugins) on GitHub.

    For building a server application we need ktor-server-core. We use Netty as web server and therefore add ktor-server-netty. To work with JSON requests/responses we use Ktor's GSON integration with ktor-gson. Ktor uses SLF4J for logging and needs a SLF4J provider, we use logback-classic.

    To see if Ktor is ready, we can run the following piece of code:

    fun main() {
        embeddedServer(Netty, 8080) {
            routing {
                get("/foo") {
                    call.respondText("Hey", ContentType.Text.Html)
                }
            }
        }.start(wait = true)
    }

    This defines a single endpoint that listens for GET requests on /foo. Running the main method starts an embedded Netty server listening on port 8080. We can now open http://localhost:8080/foo in a web browser and should see the text Hey as response.

    Configuring JSON handling

    Next we need to set up JSON handling in Ktor. We do this by adding the ContentNegotiation feature with GSON converters to our Ktor application:

    fun main() {
        embeddedServer(Netty, 8080) {
            install(ContentNegotiation) {
                gson {
                    setPrettyPrinting()
                }
            }
    
            routing { 
    			...
    		}
        }.start(wait = true)
    }

    With the install() method we can add a feature to our application. Features can provide additional functionality which can be added to the request and response processing pipeline. The ContentNegotiation feature provides automatic content conversion based on Accept and Content-Type headers. The conversion is handled by GSON.

    This way we can return a Kotlin object from an endpoint and the content conversion will be handled by Ktor. If a client accepts JSON (Accept header with value application/json) Ktor will use GSON to convert the Kotlin object to JSON before it is sent to the client.

    Implementing get operations

    In this example we create a simple CRUD API for products. For this we use two small helper classes: Product and Storage.

    Product is a basic product representation containing a name, a description and an id:

    class Product(
            val id: String,
    		val name: String,
            val description: String
    )

    Storage acts as our simplified product database, it uses a MutableMap to save products:

    class Storage {
        private val products = mutableMapOf<String, Product>()
    
        fun save(product: Product) {
            products[product.id] = product
        }
    
        fun getAll() = products.values.toList()
        fun get(id: String) = products[id]
        fun delete(id: String) = products.remove(id)
    }

    With these two classes we can now start to create our Ktor endpoints. Of course we can just add more routes to the main method we used above. However, we typically want to separate different types of routes into different files. Therefore, we create our own Route extension function, named product() in a separate file:

    fun Route.product(storage: Storage) {
        route("/products") {
    
            // GET /products
            get {
                call.respond(storage.getAll())
            }
    
            // GET /products/{id}
            get("/{id}") {
                val id = call.parameters["id"]!!
                val product = storage.get(id)
                if (product != null) {
                    call.respond(product)
                } else {
                    call.respond(HttpStatusCode.NotFound)
                }
            }
    	}
    }

    This defines two GET operations below the /products route. One to obtain all products and one to obtain a single product by id. In the single product endpoint we use call.parameters["id"] to obtain the product id from the request URI. With call.respond() we can define the response we want to send to the client.

    Next we have to create our product storage and add our routes to the server. We do this by calling our Route extension function within the routing section of our server:

    fun main() {
        embeddedServer(Netty, 8080) {
            install(ContentNegotiation) {
                gson {
                    setPrettyPrinting()
                }
            }
            routing {
                val productStorage = Storage()
    
                // add some test data
                productStorage.save(Product("123", "My product", "A nice description"))
                productStorage.save(Product("456", "My other product", "A better description"))
    
    			// add product routes
                product(productStorage)
            }
        }.start(wait = true)
    }

    We can now run this application and send a GET request to http://localhost:8080/products. We should get a JSON array containing two products as response:

    Request:

    GET http://localhost:8080/products

    Response:

    [
      {
        "id": "123",
        "name": "My product",
        "description": "A nice description"
      },
      {
        "id": "456",
        "name": "My other product",
        "description": "A better description"
      }
    ]

    Implementing create and update operations

    To create a new product, we use the POST operation on /products. Once a product has been created, we respond with HTTP 201 (Created) and a Location header which points to the newly created product. Our Ktor code looks like this:

    fun Route.product(storage: Storage) {
        route("/products") {
    		...
    	
    		// POST /products
    		post {
    			val data = call.receive<Product>()
    			val product = Product(
    					id = UUID.randomUUID().toString(),
    					name = data.name,
    					description = data.description
    			)
    			storage.save(product)
    			call.response.headers.append("Location", "http://localhost:8080/products/${product.id}")
    			call.respond(HttpStatusCode.Created)
    		}
    	}
    }

    With call.receive() we can convert the request body JSON to a Product object. Note that we use the data object from call.receive() to create another product object that is then passed to storage.save(). If we would pass data directly to storage.save() it would be possible for the client to define the product id. Like the other fields, id will be mapped from JSON into the Product object with call.receive(). However, we want to make sure we get a new server generated product id. An alternative solution would be to create a specific transfer object that does not contain the id field.

    With call.response.headers.append() we add Location header to the response.

    The following snippets show an example for a possible request/response combination:

    Request:

    POST http://localhost:8080/products
    Content-Type: application/json
    {
        "name": "New product",
        "description": "This product is new"
    }

    Response:

    201 (Created)
    Location: http://localhost:8080/products/70f01c0f-0a6f-4227-a064-02383590a5ab

    A product is updated by sending a PUT request to /products/{id}. The Ktor endpoint looks like this:

    fun Route.product(storage: Storage) {
        route("/products") {
    		...
    
            // PUT /products/{id}
            put("/{id}") {
    			val id = call.parameters["id"]!!
                val product = storage.get(id)
                if (product != null) {
    				val data = call.receive<Product>()
                    product.name = data.name
                    product.description = data.description
                    storage.save(product)
                } else {
                    call.respond(HttpStatusCode.NotFound)
                }
            }
    	}
    }

    Again we use call.parameters and call.receive() to get the product id and the content from the request body. We check if a product with the given id is available. If this is the case we update the product otherwise HTTP 404 (Not Found) is send to the client.

    Deleting products

    We delete products by sending a DELETE request to /products/{id}. The endpoint implementation does not use any new Ktor features, but for sake of completeness here is it:

    fun Route.product(storage: Storage) {
        route("/products") {
    		...
    
            // DELETE /products/{id}
            delete("/{id}") {
                val id = call.parameters["id"]!!
                val product = storage.delete(id)
                if (product != null) {
                    call.respond(HttpStatusCode.OK)
                } else {
                    call.respond(HttpStatusCode.NotFound)
                }
            }
        }
    }

    Conclusion

    Ktor provides an easy way to build a REST API with Kotlin. It uses Kotlin Lambdas to define routes and endpoints. We learned how to access request information, send JSON responses, set response headers and a few other things by building this small CRUD example service. You can find the example code on GitHub.

  • Monday, 29 June, 2020

    IntelliJ's text based HTTP client

    IntelliJ provides a HTTP client that is purely text based. While this might sound strange at the beginning it turns out that this is a very useful feature.

    Getting started

    First we need to create a file whose name ends with .http or .rest. For example: my-requests.http.

    To issue a simple GET request we have to write down the request in our newly created file.

    For example:

    GET http://localhost:8080/products

    IntelliJ now adds a small Run-Icon next to the line which allows you to execute the request.

    IntelliJ showing a run-icon next to the HTTP request line

     

    If we want to POST a piece of JSON, we simply have to add a Content-Type header and the request body:

    POST http://localhost:8080/products
    Content-Type: application/json
    
    {
      "name": "My other Product",
      "description": "hu?"
    }
    

    Please note that there has to be a blank line between headers and request body.

    Of course IntelliJ has syntax highlighting and auto completion for writing down headers and JSON:

    IntelliJ showing Syntax-completion for HTTP request headers

     

    Multiple requests in the same file need to be separated with ###. For example:

    GET http://localhost:8080/products
    
    ###
    
    POST http://localhost:8080/products
    Content-Type: application/json
    
    {
      "name": "My other Product",
      "description": "hu?"
    }

    Using variables

    With {{ .. }} we can add variables to our requests. Maybe we want to issue the same request against different environments. To support this, we can update our request with a host variable:

    GET http://{{host}}/products

    Next we need to define the {{host}} variable. For this we create a http-client.env.json file and add the following content:

    {
      "development": {
        "host": "http://localhost:8080"
      },
      "production": {
        "host": "http://my-cool-api.com"
      }
    }

    This defines two environments: development and production. Both environments define the host variable with a different value.

    When running the request, we can now choose the environment we want:

    IntelliJ showing environment in run options for HTTP request

     

    Share requests with your team

    The simple text-based request definition allows easy sharing with your team. You can even check in request files into your version control system. Of course you do not want to check in passwords or API keys that might be needed for request execution. IntelliJ supports this with a separate private environment file (http-client.private.env.json). Like in the previous environment example, we can use this file to define variables.

    For example:

    {
      "dev": {
        "api-key": "S3CR3T"
      }
    }

    To make sure no secrets are checked in, we can explicitly exclude this file from our version control system.