Saturday, February 6, 2016

Introduction to AWS presentation

My Introduction to AWS presentation that I presented at the Architecture Titans technical club.

Monday, January 4, 2016

AWS Elastic Load Balancing session stickiness

In a previous post "Configuring and testing AWS Elastic Load Balancer" I described how to configure AWS ELB to distribute load on multiple web servers.
We observed that the same client might get routed to a different EC2 instance. Some applications require the user to always be directed to the same instance during his session. This is the case when an in-memory session state is used or any other application specific reason. This requirements is often referred to as session stickiness.
AWS ELB offers two ways to provide session stickiness: using a cookie provided by the application, or using a cookie generated by ELB.

Load Balancer Generated Cookie Stickiness

Using an Expiring cookie
Using the same configuration as the previous post, the load balancer will have the stickiness configuration set to "Disabled". 
To change this behavior:
  1. Click "Edit" link
  2. Select "Enable Load Balancer Generated Cookie Stickiness" option.
  3. As a testing value, enter 60 as an expiration period.

Editing stickiness properties of ELB

Now, let's start testing the effect of this new configuration. Open the test url (for example:, check the previous post for more details). Using fiddler or network tab in developer tools of your favorite browser, you can observe that the response includes this header:

Set-Cookie: AWSELB=A703B168326729FE0B7D2675641656C1889E580D7525169B4BE36A819D9F2A18BE64415E6B0C90F5F2AC8CB0CFAF8DABE929DB27D5077D6FF616065A5BAF81DDB430BE92;PATH=/;MAX-AGE=60

As we see, it's a cookie named "AWSELB" with max-age of 60 seconds. and it applies to the whole site.

AWSELB cookie in in the response as appears in Chrome dev tools

If you refresh the page, you'll find that the browser sends the cookie as expected:

Cookie: AWSELB=A703B168326729FE0B7D2675641656C1889E580D7525169B4BE36A819D9F2A18BE64415E6B0C90F5F2AC8CB0CFAF8DABE929DB27D5077D6FF616065A5BAF81DDB430BE92

The cookie is sent by the browser, and the response does not include a new cookie

But the response does not include the cookie. So it will expire after 60 seconds and the browser will not send it after expiration. Refreshing the browser several times will direct the traffic to the same EC2 instance, we can verify this by examining the response which looks like:

Host name:
Public IP:

As soon as the cookie is still active, the request is directed to the same instance. But what happens after the max-age passes?
A new cookie will be generated and you might be directed to one of the other web servers, and a new cookie with another value will be generated by ELB:


Notice that the value of the cookie has changed. And after this cookie expires, a new cookie might be generated using the old value pointing to the first instance.

Using an ELB cookie without expiration:
If the expiration values is left blank, then the behavior differs. The cookie will be generated without the max-age value. And the browser will send the same cookie until it's closed.


What happens when a server goes down?
To try this scenario, let's shut down the server which is getting the requests and refresh the browser. This time ELB generates a new cookie pointing to a healthy instance.

ELB has a built-in mechanism to support session stickiness with no code changes from the application side.
Using an expiring cookie might not be the best option to guarantee session affinity as the cookie is not renewed and it seems that there is no way to achieve a sliding expiration window for it. So you might prefer to go with a cookie without expiration.

In the next post, we'll use the other method available for session stickiness: using Application Generated Cookie Stickiness.

Friday, May 15, 2015

Configuring and testing AWS Elastic Load Balancer

Load balancing is an essential component for the scalability and fault tolerance of web applications. Major cloud computing providers have different offerings for load balancing.
In this post I'll explore AWS's (Amazon Web Services) ELB (Elastic Load Balancing) feature, and test it to see how it distributes the load on front-end web servers, and in case of unavailability of one of the front-end servers, how traffic is directed to the healthy instance(s).

I'll use Linux based image, but the concepts apply to Windows images. I assume that the reader has the basic knowledge on how to create an AWS account and create EC2 (Elastic Compute Cloud) virtual machine. If not, don't worry, following the steps below will give you a good understanding.

So the experiment goes as follows:

1- Create a base image for front-end web servers: 


  1. Go to AWS console and select "Launch Instance", from the list of images, select "Ubuntu Server 14.04 LTS".
  2. Complete the wizard till you reach the "Configure Security Group" step. In this is the step we select the proper ports we need AWS to open. Select SSH (22) to connect to the instance to configure it, and HTTP (80) to serve web traffic.
  3. When you're prompted to select the key pair, make sure to choose an existing one you have already downloaded or create a new one and keep it in a safe place.
  4. Then Launch the instance.

Note: When I first stared using AWS, and being from a windows background, the term "Security Group" was a bit confusing to me, it's about firewall rules not security groups in the sense of Active Directory Groups.

2- Configure Apache web server

The image does not have a web server installed by default, so I'll SSH into the instance and install it.
If you're using MAC or Linux, you should be able to run SSH directly. For Windows users, you can use Putty.
  1. Copy the public IP of the running instance you just created.
  2. Use SSH to connect using this command:  ssh -l ubuntu -i . for example: ssh -l ubuntu -i mykey.pem . note that ubuntu is the username for the image we created this machine from. the .pem file acts as a password.
  3. Now we are inside the instance. It's time to install and configure Apache:

sudo su
apt-get install apache2
sudo a2enmod cgi
service apache2 restart

The above commands simply do the following:
  • Elevate privileges to run as a super user to be able to install software.
  • Install apache using the package manager.
  • Enable CGI, I'll show you why later
  • Restart apache so that CGI configuration takes effect.

Now it's time to test the web server. Visit http://INSTANCE_IP and you should be welcomed with the default apache home page.

3- Create a script to identify the running instance

To test ELB, I need to identify which instance served the request just by looking into the response to a web request. Now I have 2 options: Create static pages on each web fron-end or create some dynamic content that identifies the instance. And I prefer the latter option as I'll use the same image for all front-ends.
EC2 has a nice feature called instance metadata. It's an endpoint accessible from within EC2 instances that can be called to get information about it. From SSH terminal try:


A list of available meta-data will be shown:


Appending any of them to the URL will show the value. For example:


And I'll use these two meta-data items to identify the instances by showing them within a bash script and then serve it from apache. cd into /usr/lib/cgi-bin

cd /usr/lib/cgi-bin

This is the default location that apache uses to serve CGI content. That's why I enabled CGI in a previous step.
in that folder I'll create a bash script that shows the output of the meta-data. use any text editor. For example run nano in the command line and paste the below script:


echo "Content-type: text/text"
echo ''
echo 'Host name:'
echo ''
echo 'Public IP:'

If using nano, ctrl+X, y. save as

Now we need to grant execute permission on this file:

chmod 755 /usr/lib/cgi-bin/

To test the configuration, browse to http://INSTANCE_IP/cgi-bin/
My results look like:

Host name:
Public IP:

Note: I'm not advising using bash scripts in production web sites. It just was the easiest way to spit out info returned from the meta-data endpoints with minimal effort.

4- Create 2 more front-ends

Now we have an identifiable instance. Let's create more of it.
  1. Stop the instance from the management console
  2. After the instance has stopped, right click -> image -> create image.
  3. Choose and appropriate name and save.
  4. Navigate to AMI (Amazon Machine Image) and check the creation status of the image.
  5. Once the status is available click launch
  6. In the launch instance wizard, select to launch 2 instances
  7. Select the same security group as the one used before, it will have both 22 and 80 ports open.
  8. Start the original instance. 
  9. Now we have 3 identical servers.
  10. Using the IP address of any instance, navigate to the CGI script, for example:
Note that most probably the IP of the first instance is now different after restart.

5- Create an ELB instance

  1. In AWS console, navigate to "Load Balancers".
  2. Click "Create Load Balancer"
  3. Make sure it's working on port 80
  4. Select the same security group
  5. In the health check, in the ping path, enter "/". This means that ELB will use the default apache page for health check. In production, it might not be a good idea to make your home page the health check page.
  6. For quick testing, make the "Healthy Threshold" equal to 3.

Now a bit of explanation is required. This configuration tells ELB to check for the healthiness of a front-end instance every 30 seconds. A check is considered successful if the server responds in 5 seconds.
If a healthy instance does not respond with that period for 2 consecutive failures, it's considered unhealthy. And similarly, an unhealthy instance is considered healthy again if it responds to the check 3 consecutive times.

Now select the 3 instances to use for load balancing. And wait until the ELB instance is created and the 3 instances in the "instances" tab are shown InService.

Now in the newly create ELB, select the value of the DNS name (like and navigate to the URL of the metadata page. My url looked like:

The data displayed in the page will belong to the instance that actually served the request. Refresh the page and and see how the response changes. In my case ELB worked in a round robin fashion and the responses where:

Host name:
Public IP:

Host name:
Public IP:

Host name:
Public IP:

Inspect the network response using F12 tools and note the headers:

HTTP/1.1 200 OK
Content-Type: text/text
Date: Sat, 16 May 2015 19:12:38 GMT
Server: Apache/2.4.7 (Ubuntu)
transfer-encoding: chunked
Connection: keep-alive

Note: nothing special as there is no session affinity.

6- Bring an instance down

Now, let's simulate an instance failure. Let's simply stop the apache service on one of the 3 front-ends. So ssh into one of the 3 instances and run:

sudo service apache2 stop

Refresh the page pointing to the ELB url, note that after a few seconds, you only get responses from the 2 running instances. After about 1 minute, the instance is declared OutOfService in the Instances tab of ELB.


7- Bring it back!

This time, turn on apache service by running:

sudo service apache2 start

Wait about one and half minutes, the instance is back to InService status and you start to get responses from it.
The "Healthy Hosts ( Count )" graph shows a very good representation of what happened:

8- Turn them all off!

They are costing you money, unless you are still under the free tier. It's recommended to terminate any EC2 and ELB instances that are no longer used.

If you intend to leave some instances alive, it's recommended to de-register the instance from ELB when shut down:



In this post, we've seen ELB in action using its basic settings. The round robin load balancing worked great and health check made our site available to users by eliminating unhealthy instances.
This works great with web applications that don't require session affinity, for applications that require it, well, that's another post.

Thursday, October 30, 2014

Inside of the developer's mind

As software developers, we give special care to learning and mastering our tools, whether they are IDEs, libraries, technologies, or design patterns.
But we forget about the most important tools that we use: mind and memory. And in our continuous challenge to learn new things, our understanding of how mind and memory work is essential.
In this post I would share some knowledge about how memory works and try to explain how does this relate do our job as developers.
I’ve learned many of the concepts I’ll mention here from a great MOOC: Learning How to Learn

First, I will share with you few concepts about memory.

Human Memory

If you remember from computer science classes that you might have attended, computer systems have different types of memory. Each type of memory is used for a different type of tasks and have different characteristics.
For example registers are used for performing calculations in the processor. While hard disks are used for long term memory storage.

Surprisingly human memory works kind of the same. we have a working memory which has limited number of slots (4 to be more specific). It’s used for problem solving and to process things that require concentration like debugging and writing new code.
This memory is highly volatile. you need to concentrate to keep things active in it and stay conscious or you’ll lose them. Some times you need to repeat things few times in order for them to stick even for a short period just enough until you get to use it (imagine remembering a bug id from an issue tracking system just for a few seconds till you get to your machine). Sometimes you even close your eyes to disallow distractions from wiping it off.

On the other hand, long term memory is used to store huge amount of information but also is harder to retrieve data from. Long term memory is essential for learning, imagine that you need to learn everything from scratch again and again, you’ll not make a noticeable progress.
Same as computers work, we need to retrieve information from long term memory into the working memory to process them. So both types are connected.
So how to support your long term memory to store and effectively retrieve information when needed?

Long term memory is like a huge warehouse, where billions of things (information, memories) are stored, and these things can bury each other. To be able to retrieve them later, you need to practice retrieving them. This practice reinforces the synaptic connections that form the memory. This time, unlike Hard disks, if you don’t practice the retrieval process, you’ll not be able to find the information when you need it.
In addition, to reliably store new information, you need to give it some time so the bonds get stronger. So just repeating a piece of information several times in a short period of time (minutes or hours) won’t get it into the long term memory.
Spaced repetition is a technique used to practice the retrieval process over an increasing time intervals. For example, when you try to memorize something, you read it today, tomorrow, two days later, a week later, a month, and so on. These intervals are determined based on the difficulty of the topic and whether you’re able to recall it or not.

One great tool that employs this technique is Anki. You feed it with index cards or use a card deck from its gallery and start studying these cards. It will handle the spaced repetition, it will show you the cards that you’ve studied when it feels it’s the right time. Smart isn't it? And by the way, it has web, desktop, and mobile editions.
So the next time you find yourself unable to remember the name of a design pattern, try Anki. Some kind people contributed to the shared decks with design patterns decks. Or even better, make your own and contribute them.


Software development is fun, or so it should be. Good developers are driven by challenging problems and curiosity. But times come when we have to do boring tasks that we want to put away but we can’t as they are part of the job. What are you whispering? Writing documentation? Yes, this is a good example.
When we are required to do some tasks that we don’t like, we tend to procrastinate, find excuses to do other less important things at that time, or remember that it’s the time for a cup of coffee and some socialization.
Why? It’s because of how our brains work. When we are faced with a task or a problem that we don’t like to do, this activates areas in the brain that are associated with pain. It’s a normal reaction that brains do some tricks to avoid doing them. Thus avoid the pain and feel temporarily happier, till deadlines approach, and bosses start to get angry. Even worse, the more you procrastinate, the more painful the task gets. Similar to addiction, short term happiness, but long term loss.
What’s interesting is that if you actually start working on the task, the pain goes away! So it’s all about coming over the cue that activates the procrastination behavior.
Procrastination is a bad habit, and we want to replace it with a good habit that make us more effective. So what are the components of habits?
  1. Cue: This the trigger or event that starts the flow of the habit. For example, a distraction from social media or an email notification. Or a checking your task list to find that documentation task.
  2. Routine: The reaction to the cue. For example, finding somebody else’s task that your task is pending on in order to avoid it. Admit it, you’re good at this :)
    A good routine might be a positive self talk followed by a time-boxed duration to get things done. I’ll tell you later about a good tool for this.
  1. Reward: Maybe this is what’s dangerous about procrastination as a habit, it gives immediate reward which is removing the pain and doing something more pleasant. So key to building a good habit is to reward it, maybe with a quick technical chat with a geek like you about a hot technology. Or a cup of coffee.
  2. Belief: Our beliefs about what’s good and what’s bad, what’s useful and what’s not. To tackle procrastination, you need to understand and believe that it has a negative long term effect.

To put this into action, Pomodoro is a very effective technique of handling procrastination. Pomodoro is the Italian word for tomato . This is how it works:
  1. Disable any interruptions. (No unwanted cues)
  2. Focus on a task for a short period, 25 minutes by default, which is not too hard. As I mentioned before, once you start, the pain goes away. Use the timer for this.
  3. Give yourself a 5 minutes break as a reward. You can get your cup of coffee now. Again, use the timer.
  4. Start the next 25 minutes of focused work and repeat.
  5. Give yourself a longer break after say 4 pomodoros.
This technique is very useful and proven. You can find mobile apps that employ this technique to help you manage your pomodoros and track them.

A pomodoro timer, from

In addition, you need to focus on the process of the task, rather than focusing on the outcome (the product) that triggers the pain.
Try to get into the flow mode which gets the best out of your concentration and focus. Have a look on this talk to know more about it.

Mark Seemann - Zone out, check in, move on from NDC Conferences on Vimeo.

Focused and Diffuse modes of thinking

We’ve all had time struggling against a weird bug or trying to find a solution for a complex problem, but no good. Trials after trials, hours after hours, Googling, tweaking, and squeezing your mind to the last drop of good ideas, and it all seems as if you’re slamming your head in the wall.
We’re all about focus, but focused mode of thinking won’t always get you where you want. It sometimes limits the space of your brain where you’re searching for the solution. It keeps finding the ideas that you’re familiar with, which might be the reason why you’ve got into this buggy situation in the first place. Or the reason why you can’t figure out the best design for this novel problem. Similar to how greedy algorithms sometimes can’t find the optimum solution by being stuck into a local minimum or maximum.

Local maximum and minimum. From
So, should you give up? No, you should just switch to a different mode of thinking , the diffuse mode, which allows you to explore wider solution spaces and form new connections in your brain that finally formulate the solution you’re looking for. How to do the switch? It’s easy. Get away from the computer’s screen, leave your desk, have a walk and take a breath in the fresh air. Do this while giving yourself a chance to think about other things that might not be directly related to your problem. Looking into the big picture and seemingly unrelated areas will explore wider areas of your brain’s memory and will help you getting the solution.

Note that these two modes of thinking cannot be used simultaneously. You cannot be in both states at the same time. So you might need to go back and forth between focused and diffuse modes in order to find an innovative solution or understand a difficult concept.


Understanding how your brain works is essential to your career as a software developer. Ability to learn and understand new concepts and use that understanding in the right place and time is a cornerstone in your arsenal of skills.

Thursday, August 21, 2014

Code disclosure pattern issue with AjaxControlToolkit

I was helping a team troubleshooting some issues found when they ran IBM Security AppScan tool against a web application.
One of the issues reported by the tool was:

Web Application Source Code Disclosure Pattern Found

Severity: Medium
URL: URL/ScriptResource.axd
Entity: ScriptResource.axd (Page)
Risk: It is possible to retrieve the source code of server-side scripts, which may expose the application logic and other sensitive information such as usernames and passwords
Latest patches or hotfixes for 3rd. party products were not installed
Temporary files were left in production environment
Debugging information was left by the programmer in web pages
Fix: Remove source code files from your web-server and apply any relevant patches

And the report included the following sample response:
// Add common toolkit scripts here. To consume the scripts on a control add
//public class SomeExtender : ...
// to the controls extender class declaration.

So the highlighted part indicated to appscan that the application exposed some code. And as I have previous experience with AjaxControlToolkit I could give the proper advice.

Referring to the source code of AjaxControlToolkit, you can find that the file Common/Common.debug.js includes the above lines as a hint to developers. So it sounds harmless, another false positive, and we can live with it, right?
Probably not. The release version of the above file Common/Common.js does not include any of this C# code. This is an indicator that a debug version is being used in the test. What started as a security issue, is now a performance issue, and using release build should fix it.

In general, automated security, code analysis, or performance testing tools generate false positives, but they should not be ignored without proper analysis.

Note: the latest version of AjaxControlToolkit includes the commented hints in the Client/MicrosoftAjax.Extended/Common/Common.pre.js file

Tuesday, July 29, 2014

Do's and Don'ts of Agile Retrospectives article on

Team retrospectives is an important Agile practice, check my article Do's and Don'ts of Agile Retrospectives to find some tips about having a healthy retrospective and avoiding common pitfalls.

Friday, July 18, 2014

Using Azure command line tools in Windows batch files

I have an Azure IaaS environment for development and experiments. The environment consists of 3 machines. I wanted to automate starting and shutting down the VMs in this environment.
I used Azure Command-Line Interface to create two batch files, one to start the environment and the other to stop it.

The scripts looks like:

azure vm start VMName1
azure vm start VMName2
azure vm start VMName3


azure vm shutdown VMName3
azure vm shutdown VMName2
azure vm shutdown VMName1

But when I executed the batch files, only the first command is executed then the script just terminates even if the command is successful. I had no clue why!!

I asked my friend @mShady who thankfully pointed me to this stack overflow thread: The answer tells: "you must call another script using the call command so it knows to return back to your script after the called script completes. Try prepending call to all commands."

So I added call to the batch files:

call azure vm start VMName1
call azure vm start VMName2
call azure vm start VMName3


call azure vm shutdown VMName3
call azure vm shutdown VMName2
call azure vm shutdown VMName1

And it worked.

The documentation for call mentions that it "Calls one batch program from another without stopping the parent batch program".

After checking, I found that the azure command is a batch file that internally calls node. And that was the issue.
The path of the batch file (which is in the path environment variable to be visible anywhere) is: "C:\Program Files\Microsoft SDKs\Windows Azure\CLI\wbin\azure.cmd"