Dominic Maas

SoundByte: Improving HTTP Requests

How I greatly improved HTTP request speed within SoundByte using just three lines of code.



SoundByte heavily relies on accessing data from the SoundCloud & Fanburst API. When I first started work on SoundByte a year ago, optimisation was not a priority for me, I just needed to get a base app working as a proof of concept. But as the app grew the time taken to perform API requests (especially multiple requests at once), these requests started adding up, slowing down the entire app.

In order to speed up these requests, I did two things, first I enabled compression (and decompression) and second I enabled the ability to parse data as it is streamed to the client. 

To begin with, I had this un-optimized code:


// Create HTTP client within a 'using' clause to auto dispose resources.
using (var httpClient = new HttpClient())
{
    var json = httpClient.GetStringAsync(requestUri);
    var data = JsonConvert.DeserializeObject<API.Endpoints.Track>(json);
    // Loop through results and add to List View/Grid View.
}
                        

At first, nothing looks wrong, but what happens when the user has no internet access? Or what happens when the API changes? If any of these things happen, the app will crash, resulting in poor user experience. The user will be left confused. Another problem with this code is that it is just plain slow. There is no HTTP compression, and we are saving the entire body of the response, then deserializing it.

A useful feature of the HttpClient constructor is that it allows you to pass in a HttpClientHandler class. This class allows you to change many default functions of the HttpClient, but we will be focusing on one for today, 'AutomaticDecompression'. The AutomaticDecompression method is a simple way to add the 'Accept-Encoding: gzip, deflate' header to the HTTP Request. This will also allow the HttpClient to take care of decompression for you.

If we add some of this extra code, we now get:


// We create the handler, check if the device supports automatic decompression
// and then specify which decompression methods we want to use.
var handler = new HttpClientHandler();
if (handler.SupportsAutomaticDecompression)
{
    handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
}

// Create HTTP client within a 'using' clause to auto dispose resources.
// Pass in the HttpClientHandler
using (var httpClient = new HttpClient(handler))
{
    var json = httpClient.GetStringAsync(requestUri);
    var data = JsonConvert.DeserializeObject<API.Endpoints.Track>(json);
    // Loop through results and add to List View/Grid View.
}
                        

Just by adding these few lines into my app, I experienced a sudden gain in performance which contributed to about a quarter of my 60% speed increase.

But we can go further. At the moment we are converting the entire request to a string, deserializing that string, and then only using the deserialized object, discarding the string. This is very inefficient.

In the next tutorial I will talk about how I further decreased SoundByte load times by deserializing the HTTP stream on the fly.

Dominic Maas
11 June 2017

SoundByte: Scaling Up

Dealing with the SoundCloud API limitations while scaling up SoundByte.



What I'm about to show and tell you is not pretty. It had to be done in order to scale up SoundByte in order to handle increased demand. But first lets talk about the SoundCloud API. Working with the SoundCloud API is a very love-hate type of releationship. At times the API is lovely to work with, endpoints just work, I can grab the information I need quickly.

Other times it's a nightmare, staying up till 4am on a school night desperately trying to email SoundCloud about a stupid limitations, while at the same time people are complaining that an app that they bought is not working, and blaming me for it. As I said, love-hate.

This one limitation that caused by so much pain, was the SoundCloud Playback Rate Limit. This rate limit limited the amount of songs that can be played by my app every 24 hours. (At this time, 15000 requests). There is no way to increase the limit.

At first you may think, 15,000 request? 15,000 songs played, that's plenty, why are you having issues?

Well, 15,000 may be a lot, but daily (24 hours) SoundByte averages at around 16,000 - 20,000 song changes (with a recorded peak of 30,000 song changes in a day). That's 130,000 song changes per week over 1,300 users.

15,000 does not look so large anymore does it?

16,000 to 20,000 song changes does not directly relate to the rate limit either, so this number is actually a lot higher when looking at the rate limit side of things. The reason for this is that SoundByte caches the entire playlist before playing (to improve user experience with gapless playback). These playlists average around 80 songs in size, and may or may not all be listend to.

Earlier this year I tried to contact SoundCloud about increasing the API limit for SoundByte. The hate part of the relationship appeared once again:

So I took matters into my own hands. When I present is currently used within SoundByte, and technically breaks the SoundCloud Developer Terms and Conditions, but they gave me no option.

The general idea is prior to loading a playlist, I first determine if the API key I want to use is active. (I do this by pinging a custom track I uploaded onto SoundCloud - fun fact, this track now has 60,000 'plays'). If the API key is active, I use to to load a playlist, if it is inactive, I perform the same steps to the next API key.

The code for this can be seen below:


/// <summary>
/// Goes through the primary and backup keys to determine which key is best 
/// to use for generating the playlist.
/// </summary>
/// <returns>The API Key to use for this playlist.</returns>
public async Task GetCorrectApiKey()
{
    // The client ID to use
    var clientId = SoundByteService.Current.SoundCloudClientId;

    // Base track uri
    var uri = "https://api.soundcloud.com/tracks/320126814/stream?client_id=";

    // Check if we have hit the soundcloud api limit
    if (!await SoundByteService.Current.ApiCheck(uri + SoundByteService.Current.SoundCloudClientId))
    {
        // Try go through the backup keys
        if (await SoundByteService.Current.ApiCheck(uri + SoundByteService.Current.PlaybackIDs[0]))
        {
            clientId = SoundByteService.Current.PlaybackIDs[0];
        }
        else if (await SoundByteService.Current.ApiCheck(uri + SoundByteService.Current.PlaybackIDs[1]))
        {
            clientId = SoundByteService.Current.PlaybackIDs[1];
        }
        else if (await SoundByteService.Current.ApiCheck(uri + SoundByteService.Current.PlaybackIDs[2]))
        {
            clientId = SoundByteService.Current.PlaybackIDs[2];
        }
        else if (await SoundByteService.Current.ApiCheck(uri + SoundByteService.Current.PlaybackIDs[3]))
        {
            clientId = SoundByteService.Current.PlaybackIDs[3];
        }
    }

    return clientId;
}
                        

This code is not ideal, first of all it does not take into account the 'in-between' periods. This means that playback ID 1 may still be valid for 5 more plays, a user generates a playlist of 50 songs. Only 5 of these songs will be played.

Future versions of SoundByte aim to directly count how many requests are made to this rate limited API. This allows for easier prediction and analysis of how make service keys I'm going through. I'm also aiming to greatly improve efficiency when playing songs within SoundByte. Currently if you start playlist a playlist and then choose the shuffle the playlist, the tracks are generated fresh each time. So if a playlist has 50 tracks. 50*2 + (2 checks) = 102 hits to this rate limited API. The user then might listen to one song, and close the app.

Future versions of SoundByte introduce more aggressive caching to make sure songs are loaded once and only once per playlist.

Dominic Maas
7 June 2017

JQuery: Creating an image slider

How to easily create and implement a JQuery image slider with basic programming knowledge.



Creating a image slider on your website is very easy using JavaScript and JQuery. In this tutorial I'll show you how to setup your development workspace, link the required files, download and use JQuery and then create your image slider.

Here is a picture of what we are going to create:

Image Slider Example

Setup:

First we must setup your development workspace. If you already have an existing site, skip the the 'intergration' section below. I personally recommend using Visual Studio Code or Notepad++ for text editing. Once installed create a new folder where you want to save your website. I'll create one in OneDrive called 'Website'.

Within this folder create two folders, once called styles, the other called scripts.

Create a file called index.html. This will be the core of your website. Within the styles folder create a new file called stylesheet.css, and within the scripts folder create a new file named script.js.

You should have the following folder layout:


.
├── styles
|   └── stylesheet.css
├── scripts
|   └── script.js
└── index.html
                        

In order to continue with this tutorial, you will need some basic code with your stylesheet and index documents. Open up these files in your favourite text editor and paste in the following.

index.html


<!DOCTYPE html>
<html>
    <head>
        <!-- Page title -->
        <title>Test Image Slider</title>
        <!-- Link your CSS -->
        <link rel="stylesheet" type="text/css" href="styles/stylesheet.css">
    </head>

    <body>
        <!-- Your content will go here -->
    </body>
</html>
        
                        

stylesheet.css


html, body { margin:0; }

body 
{
    /* Use a sans serif font, start with Windows 10 default */
    font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
    /* Gradient background */
    background: linear-gradient(-90deg, #F1F1F1 , #E1E1E1);
}
                        

You are now ready to continue with the rest of the tutorial!

Creating the Image Slider:

To start with this tutorial we must first download JQuery. JQuery is a JavaScript library that greatly improves programming ability within JavaScript and generally makes life a lot easier. Adding JQuery into your website can be done in two ways. You can manually download JQuery, place it into your scripts folder and then link it within your html file. Or you can use the free CDN. In this tutorial I will be using the CDN.

Add this code to the bottom of your index.html page. It is generally recommend to place scripts at the bottom of your webpage to increase page load time.


        ...
        <!-- JQuery Script hosted from the Microsoft ASP.NET CDN. -->
        <script src="http://ajax.aspnetcdn.com/ajax/jQuery/jquery-3.2.1.min.js"></script>
        <!-- Your script -->
        <script src="scripts/script.js"></script>
    </body>
</html>
                        

Now that this script is added, create an images folder within your working directory and place some images inside the folder. The name of these images and their extension is important. Remember them.

Next up we need to add some code to our HTML file. This code will be the basics for our image slider


        <!-- Main slider content, the background for this element will be changed -->
        <div class="slider-content">
            <!-- Button that moves the slideshow forward -->
            <a class="button next-button" href="#">Next</a>

            <!-- Button that moves the slide show back -->
            <a class="button back-button" href="#">Back</a>

            <!-- Info Text -->
            <p class="slider-text"></p>
        </div>
                        

With the HTML now added, we need to update the stylesheet.css file with the following code.


.slider-content
{
    /* Center the background image, scale it up / 
      down to make sure it fits and looks nice. */
    background-position: center;
    background-repeat: no-repeat;
    background-size: cover;

    /* Layout of image slider, 50% of web browser height, 
       1000px wide, 50px from top and centred. */
    height: 50vh;
    width: 1000px;
    margin-left: auto;
    margin-right: auto;
    margin-top: 50px;

    /* Fluent based shadow. */
    box-shadow: 0 18px 49px 0 rgba(46,89,81,.87);

    /* Enable the animation */
    transition: background 0.5s ease;

    /* Enable positioning of content */
    position: relative;
}

.slider-text
{
    /* Position at the bottom of the slide show element */
    position: absolute;
    left:20px;
    right:20px;
    bottom:-40px;
    
    /* Center align the text */
    text-align:center;
    
    /* Apply fluent design */
    background-color: rgba(255,255,255,0.9);
    box-shadow: 0 18px 49px 0 rgba(46,89,81,.27);

    /* Add some breathing room */
    padding:15px;
}

.button
{
    /* Position at the bottom of the image slider */
    position:absolute;
    bottom:-8px;

    /* Show above caption text */
    z-index:20;

    /* Remove default link styling */
    color: #333;
    text-decoration: none;
}

/* Set this button 40px from the right. */
.next-button { right: 40px; }

/* Set this button 40px from the left. */
.back-button { left: 40px; }
                        

And now the real stuff. Add the following javascript to your script.js file. This code is commented to help you understand how it works.


// An array of images that you want to 
// use on this site.
var images = ["https://designcontest-com-designcontest.netdna-ssl.com/blog/wp-content/uploads/2016/06/1be9d930234015.56197d97e0679.png", 
              "http://blenderartists.org/forum/attachment.php?attachmentid=439983&stc=1",
              "https://s-media-cache-ak0.pinimg.com/originals/58/03/ba/5803ba125d0937d1ad7e7f630ee04118.png"];

// An array of captions. Must be in the same 
// order as the images.
var captions = ["This ia a picture of a low poly desert with some buildings.", 
                "A low poly deer in the woods.", 
                "Low poly island with water"];

// The current position that we are in the 
// array.
var currentIndex = 0;

// Is the slide show locked
var isLocked = false;

// Called when the page is loaded
$(function() {
    // Start the slide show right away
    nextSlide();

    // Complete this task every 5 seconds
    setInterval(function() {
        // If the slideshow is locked, do nothing
        if (isLocked)
            return;
        
        // Go to the next slide
        nextSlide();
    }, 5000);
});

// Called when the next button is pressed
$(".next-button").click(function() {
    // Go to the next slide
    nextSlide();

    // Lock the auto slideshow
    lockSlideShow();
});

// Called when the back button is pressed.
$(".back-button").click(function() {
    // Go to the previous slide
    previousSlide();

    // Lock the auto slideshow
    lockSlideShow();
});

// Go to the next slide
function nextSlide() {
    // Increase the current index
    currentIndex++;

    // If the index is above the array limit
    // set it to zero.
    if (currentIndex > images.length - 1)
        currentIndex = 0;
    
    // Change the content
    changeContent();
}

// Go to the previous slide
function previousSlide() {
    // Decrease the current index
    currentIndex--;

    // If the index is below zero, go to the 
    // last item
    if (currentIndex < 0)
        currentIndex = images.length - 1;

    // Change the content
    changeContent();
}

// Update the content, in this case it
// involves setting the background css element.
function changeContent() {
    // Set the slider content css
    $(".slider-content").css("background-image", "url('"+images[currentIndex]+"')");
    // Change the caption
    $(".slider-text").html(captions[currentIndex]);
}

// This is called when the user manually changes the slide. 
// We pause the slideshow for 10 seconds.
function lockSlideShow()
{
    // Lock the slide show
    isLocked = true;

    // After 10 seconds unlock the slide show
    setTimeout(function(){
        isLocked = false;
    }, 10000);
}
                        

And your done! If you want to see this code in a real live example, check the main page of my website, it makes use of a very similar system.

Dominic Maas
10 June 2016