Overcoming IO Overhead

Assume there are 2 services

• Service A
• Service B

Both of these are online. When Service A makes a request to Service B, it has to wait until Service B responds before proceeding with its next instructions. To go into further detail Service A will run on a thread and from this thread a request will be made to Service B and until we get the response back this thread will keep waiting

This model is called as one-thread-per-request model

This is an expensive operation because only one thread can run on a CPU core at any given point of time. So if a thread is waiting for a response back then other threads also have to wait for this thread to get off the CPU core so they can do their tasks

There are two ways of solving this problem

• User Space Threads / Lightweight Threads
• Asynchronous Programming
  • Thread Based
  • Event Based

User Space / Lightweight Threads

Instead of using the Kernel Threads (ie threads managed by the OS which run on the CPU core directly) the plan is to use a "user space" thread that runs on our language model/framework. In other words our application will have multiple threads, one that makes a request to Service B and another that does something else and so on but ALL of them will be mapped to a single Kernel Thread

So for example if we have 5 Kernel Threads then each kernel thread could be mapped to 20 user space threads. If one of the user space thread is going to have to wait for some other process then our language/framework can swap it out with another user space thread

The Kernel thread itself won't have any clue that a user space thread changed above it, because all it sees is a single process running since the language model/framework code is handling it

The Context Switches are a lot cheaper too. We basically would have 100 user space threads running on 20 kernel threads which are in turn running on 4 CPU cores forming a funnel shaped system

Ideally this means that to achieve concurrency the programmer only needs to learn the language/framework and does not need to worry about how it works under the hood

Some examples of frameworks/languages that use this model are

• Quasar Framework (Java)
• Coroutines in Go - Goroutines
• Coroutines in Kotlin

Asynchronous Programming

The idea here is that threads submit their tasks for execution and move on to other tasks and then only return to their original tasks when they get notified that it has been completed and that it can be taken up by the thread again.

Thread Based Model

Also called as the work stealing model. This involves having a group of thread pools each of which specialize in a specific kind of task. When a thread has a task it will hand it over to the corresponding thread pool for a thread there to take it up while this thread can go and do something else. There might be some blocking within the threads in the thread pool but other than there is no blocking in the actual thread where our application tasks are running

Event Based Model

Here a thread starts off a task in the kernel and stashes away the call stack and then does its other tasks and once the task is done it is notified that it is done after which it can come back and pick the task up