Using Cloud Watch Agent to Log messages from EC2 DotNet webApi to AWS CloudWatch

 So you have your dotnet webapi. And this api runs on AWS EC2 instance and produces logs. It will be nice that you don't have to login into the EC2 with some remote client like putty, but see the logs directly in the CloudWatch UI provided by AWS. 

Why do you want to send it to the CloudWatch? Because AWS provides you with a lot of tools that allows to query the CloudWatch logs in the friendly way (for example with SQL ) and display the results in the dashboard.

By default, when you write application logs, they are written to the location you specify and nothing more. You can also use AWS SDK to write application logs directly to CloudWatch. But today you run your instance with AWS and tomorrow you decide that you want to move the application to another cloud. But what about all the code that actually writes the logs to CloudWatch. It becomes irrelevant. 

The solution is to use the CloudWatch Agent. It is AWS utility that runs on the EC2 and can monitor log files and once it has a new content, it automatically sending the content to CloudWatch. Let me show you how to use it. 

What I am not going to explain here:

1. How to start the EC2 instance 

2. How to develop dotnet webapi

So I created very basic dotnet webapi. (You know the "weather" template).

The only thing I added is "Serilog" logging. (If you didn't use Serilog in you .Net application, you probably first year student or just emigrated from Mars).

And I put some custom log message when accessing the "weather"  end point

EC2 instance I am running is Centos 7. I followed this Microsoft guide to install the dotnet runtime (I used dotnet 3.1 release)

I started the application:

and tested that it is accesible

I checked that Serilog produced a log file and indeed it was there

So far so good. Now we need to install the agent

sudo yum update -y

sudo yum install -y awslogs

Modify "/etc/awslogs/awslogs.conf"

Change the content to point to the same region where you started the EC2 instance.

sudo yum install amazon-cloudwatch-agent

Now you need to configure the agent.

run "/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard" command.

It will open some configuration wizard. You can accept all the defaults till the point the wizard will ask:

For the log file provide the full path to the dotnet application log file and also specify the log group name. It will be used to identify your logs in CloudWatch.

As the last output of the wizard you will see the location of agent configuration file

You will need this path because the next command will actually start the agent by using this path as a parameter 

 /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name

Almost done. All of the above will not work if your EC2 instance doesn't has a role that allows to write to the cloud watch. Go ahead and create the role with the following policy and attach it to EC2

Now access "weather" url several times to generate some logs (By the way forgot to mention that you need to open port 5001 in EC2 security group tp access application end-point externally).

So lets check the CloudWatch interface for the logs.

Yes!!! We see the logs.