Using AWS Elastic Search with Serilog in .Net and inspecting the actual Message

 AWS provides a managed service for Elastic Search + Kibana for everyone who wants (and can pay 😀)

Bellow, I will provide an example about how to write a log to Elastic and also to log the actual HTTP request/response from Elastic.

For the logging, I will use the very famous Serilog library that allows logging the message to different outputs (Console/File/ElasticSearch....) very easily.

Create a new console project and add the following Nuget packages.

The logon purposes you need to have an AWS account and to know all the credentials. See code inline comments to understand the purposes

class Program

    {
       
        //capture the response/request to the console
        public static void RequestCompleted(IApiCallDetails le)
        {
            Console.WriteLine(le.DebugInformation);
        }

        static void Main(string[] args)
        {
            //Logging Credentials/Data
            Environment.SetEnvironmentVariable("AWS_ACCESS_KEY_ID", "XXXXXXXXX");
            Environment.SetEnvironmentVariable("AWS_SECRET_ACCESS_KEY", "XXXXXXXXXXXXXXXXX");
            Environment.SetEnvironmentVariable("AWS_REGION", "eu-central-1");

            //Tell Serilog to write messages to Elastic
            Log.Logger = new LoggerConfiguration()
            .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("https://somehost_in_aws"))
            {
                BatchPostingLimit = 1,
               
                ModifyConnectionSettings = conn =>
                {
                    var httpConnection = new AwsHttpConnection("eu-central-1");
                    var pool = new SingleNodeConnectionPool(new Uri("https://somehost_in_aws"));
                    var conf = new ConnectionConfiguration(pool, httpConnection);
                    
                    //Show the message pritty printed
                    conf.EnableDebugMode();
                    conf.PrettyJson();
                    //this is the callback that will handle printing of the request/response
                    conf.OnRequestCompleted(e=> RequestCompleted(e));

                    return conf;
                },
                //the name of the index (some kind of the table) that will be created in Elastc
                IndexFormat = "itest-{0:yyyy.MM}",
            })
            //We also want to see the messages in the console
            .WriteTo.Console()
            .CreateLogger();

            //this message will be sent to the console and also to Elastic
            Log.Logger.Error("Error Message from Demo");
           
            Console.ReadKey();

        }
    }

After we run the program in the console we will see the actual request/response

Api call with long running asynchronous backend with .Net core - Part2

 In the previous part, I explained how to wait for synchronous operation in the backend. In part2 I will explain how to not to wait for the long-running operation, but query the API to get the information if the client needs to wait more or the request is over.

It will work in a way that when you invoke the API it can return the information immediately or it can return custom code 301 with a header "Reply-After" which holds how many time the client should wait before retrying API query. In this case, it will be the responsibility of the client to run the query again to check if the operation is over.

Http header "Reply-After" is explained in the article here

I will post the changes I did in the frontend API and backend API. It has the comments that explain what is the purpose of each function.

Frontend API

/// <summary>
        /// If initial request "long-running-async" didn't complete 
        /// immediatelly, it means that client for some token that he can use to check 
        /// if the job is done or not. The client is checking this by using "long-running-async-check"
        /// method
        /// </summary>
        /// <param name="token"></param>
        /// <returns></returns>
        [HttpGet]
        [Route("long-running-async-check/{token}")]
        public async Task<IActionResult> RunLongOperationAsyncCheck(string token)
        {
            //for the simplicity of the process we will use 
            //get operation of HttpClient object
            var client = _clientFactory.CreateClient();
            return await GetBackendResponse(token, client);

        }
        /// <summary>
        /// Service method to contact the backend and check
        /// if the operation is over or not
        /// </summary>
        /// <param name="token"></param>
        /// <param name="client"></param>
        /// <returns></returns>
        private async Task<IActionResult> GetBackendResponse(string token,HttpClient client)
        {
            var result = await client.GetAsync($"http://localhost:5010/BackendAPi/run-backend-op-async/{token}");
            var resultContent = await result.Content.ReadAsStringAsync();

            //if operation is not over yet, send the response which is asking the 
            //client to retry after specific amount of time
            if (result.StatusCode == HttpStatusCode.MovedPermanently)
            {
                Response.Headers.Add("Retry-After", resultContent);
                Response.StatusCode = 301;

                return new JsonResult(token);
            }
            //otherwise , just return the result
            return Ok(resultContent);
        }

        /// <summary>
        /// initial entry point to start long running operation.
        /// It could be that the operation on the backend is fast 
        /// and in this case the method can return with a correct response.
        /// But it could be that the backend return "Hei, the operation will take some time,
        /// you can use the following token to check the status. You don't need to check every nanosecond,
        /// I will provide you how much time do you need to wait by using Reply-After header"
        /// </summary>
        /// <returns></returns>
        [HttpPost]
        [Route("long-running-async")]
        public async Task<IActionResult> RunLongOperationAsync()
        {
            //for the simplicity of the process we will use 
            //get operation of HttpClient object
            var client = _clientFactory.CreateClient();
            //this will be unique identifier of the long running job
            Guid guid = Guid.NewGuid();
            return await GetBackendResponse(guid.ToString(), client);



        }

Backend API

 /// <summary>
        /// This is a simulation long running process that informs
        /// the client of operation is over or he needs to wait sometine
        /// </summary>
        /// <param name="token"></param>
        /// <returns></returns>
        [HttpGet]
        [Route("run-backend-op-async/{token}")]
        public IActionResult RunBackendOpAsynch(string token)
        {
            //I use the dictionary to simulate the decision making.
            //First time the backend gets the job and it decides 
            //that he needs to wait m he will put the information in the dictionary
            //so it will know that the job is not done yet.
            //Once the job is done , the token will be removed from the dictionary
            var tmp = rnd.Next(1, 5).ToString();
            
            
            //For the purpose of the simulation, if the result of random calculation is 1
            //backwend will assume that the job is done
            if (rnd.Next(1, 4) == 1)
            {
                dict.TryRemove(Guid.Parse(token), out _);
                return Ok($"Result of long running {rnd.Next()}");
            }
            else
            {
                //if random result is not 1, backend will assume that the job is not done yet
                Response.Headers.Add("Retry-After", tmp);
                Response.StatusCode = 301;
                dict.AddOrUpdate(Guid.Parse(token), tmp, (key, oldValue) => tmp);
                return new JsonResult(tmp);
            }

                 
            
            return Ok($"Result of long running {rnd.Next()}");


        }

This time I used Postman to run the test since browsers don't treat status 301 well

As we can see we got the proper result. Postman does the simulation of the client asking to check every X seconds (provided in the Reply-After header) if the job is done or not

Link to solution files

Api call with long running asynchronous backend with .Net core - Part1

 Hello friend. Let's assume that you expose some API (in our case REST API) to the client. The API doesn't actually do anything useful. It is just an entry point to a more complex backend operation. So since the API gets the operation if call the backend and return the result to the caller. The single flow looks like this.

Simple from the client to the backend with 

On the backend, you have some operations. This operation can be very fast and in this case, the client will not notice at all the round trip that API did to the backend. But it also can be that the backend will run some long operation. In this case, it doesn't make sense to keep the client waiting for the long-running operation to complete. Also, we need to remember that client invoked the HTTP call to API and HTTP call can have a timeout. It is also possible that the API is protected by external services like Cloudflare that also has a default timeout. 

A more serious problem is the fact the back end is actually asynchronous and once it is invoked from the API, the API no longer has a stateful connection that he can monitor and know if the process ended or not. 

Bellow, I will propose two possible solutions to the problem. 

One is the option to accept the connection from the client, run asynchronous operation, wait for it to complete, get the result and return it on the same connection that the client opened. 

The second option is to get the request from the client, invoke the asynchronous operation, and return immediately to the client with a message, how many times the client should wait till he needs to check if there is a response ready. In this case, it will be the responsibility of the client to check for a response again.

We will first of all focus on the first option. For this, I will create 2 web API services. One will simulate API service and the second one will simulate the backend. Both services will be .Net core web API projects.  I will put pieces of the code with a inline comments and also a link to download the project

Front end API code

    /// <summary>
    /// this is out sample model
    /// </summary>
    public class ApiModel
    {
        public ManualResetEvent ResetEvent { set; get; }
        public string ApiResult { set; get; }
    }

Front end controller

[ApiController]
    [Route("[controller]")]
    public class FrontendApi : ControllerBase
    {
        private readonly IHttpClientFactory _clientFactory;
        //this dictionary will hold the information about long running jobs
        private static ConcurrentDictionary<Guid, Model.ApiModel> dict = new ConcurrentDictionary<Guid, ApiModel>();
        private readonly ILogger<FrontendApi> _logger;
        /// <summary>
        /// Constructor with dependancy injection
        /// </summary>
        /// <param name="logger"></param>
        /// <param name="clientFactory"></param>
        public FrontendApi(ILogger<FrontendApi> logger,
                           IHttpClientFactory clientFactory)
        {
            _logger = logger;
            _clientFactory = clientFactory;
        }

        /// <summary>
        /// End point that gets the result from the backend and release the Wait
        /// </summary>
        /// <param name="guid"></param>
        /// <param name="result"></param>
        [HttpGet]
        [Route("long-running-result/{guid}/{result}")]
        [ApiExplorerSettings(IgnoreApi = true)]
        public void  RunLongResult(string guid,string result)
        {
            //the GUID we generated at the beggining of the process
            //is being passed back from the backend together with a result.
            //we use this guid to identify the data in the dictionary and 
            //relese the Wait by using apiModel.ResetEvent.Set(); command
            bool isGuidCorrect = Guid.TryParse(guid, out Guid operationId);

            if (isGuidCorrect)
            {
                if (dict.TryGetValue(operationId, out ApiModel apiModel))
                {
                    apiModel.ApiResult = result;
                    //release waiting event
                    apiModel.ResetEvent.Set();
                }
            }
        }
        /// <summary>
        /// Entry point of the example. Call long running process on the backend
        /// and wait till it is finished
        /// </summary>
        /// <returns></returns>
        [HttpPost]
        [Route("long-running-op")]
        public async  Task<IActionResult> RunLongOperation()
        {
            //for the simplicity of the process we will use 
            //get operation of HttpClient object
            var client = _clientFactory.CreateClient();
            //this will be unique identifier of the long running job
            Guid guid =  Guid.NewGuid();

            await client.GetStringAsync($"http://localhost:5010/BackendAPi/run-backend-op/{guid}");
           
            //add informaton about the job we are waiting for to the dictionary
            //we use ManulResetEvent that will wait for the signal to process the job
            ManualResetEvent mre = new ManualResetEvent(false);
            dict.TryAdd(guid, new ApiModel()
            {
                ResetEvent = mre
            }) ;
            //if no response after 10 seconds - continue anyway
            mre.WaitOne(1000 * 10);

            //handle the response
            if (dict.TryRemove(guid, out ApiModel apiModel))
            {
                if (!string.IsNullOrEmpty(apiModel.ApiResult))
                    return Ok(apiModel.ApiResult);
                else
                    return BadRequest("No result");
            }
            else
            {
                return BadRequest("Missing model object");
            }
            
        }
    }

Backend API

  /// <summary>
    /// Controller that simulates long running operation
    /// </summary>
    [ApiController]
    [Route("[controller]")]
    public class BackendAPi : ControllerBase
    {
        private static Random rnd = new Random();
        private readonly IHttpClientFactory _clientFactory;
        private readonly ILogger<BackendAPi> _logger;

        public BackendAPi(ILogger<BackendAPi> logger,
                          IHttpClientFactory clientFactory)
        {
            _logger = logger;
            _clientFactory = clientFactory;
        }

        /// <summary>
        /// Out long running operation is very simple
        /// we just wait for 5 seconds and after this 
        /// send the randon number back to the frontend API
        /// </summary>
        /// <param name="token"></param>
        /// <returns></returns>
        [HttpGet]
        [Route("run-backend-op/{token}")]
        public IActionResult RunBackendOp( string token)
        {
            //simulation of the long running task 
            Task.Run(async ()=>
            {
                await Task.Delay(1000 * 5);
                var client = _clientFactory.CreateClient();
                var res = $"Result of long running {rnd.Next()}";
                await client.GetStringAsync($"http://localhost:5020/FrontendApi/long-running-result/{token}/{res}");
            });
            //note that the call from the fronend returned immediatelly, so frontend
            //can process another requests, but the actually result will be returned a little bit 
            //later
            return Ok("I got the token and started long running operation");
        }
    }

I also added swagger UI so you don't need to use Postman in order to run the test

Don't run the project on IIS. Start it on Kestrel

In the next part, I will explain the second option (asynchronous call). 

Link to solution

Connecting to sap business one from c#

What do you need to do in order to connect to sap business one from c#?

First of all you need to know which version of the API do you want to use and does API is 32bit or 64bit. 

The API is called "DI API" and can be found on the same server where you did the SAP installation in the same directory.

For example, DI API version 100 is located in

Before you actually can use the API you need to register the SAP Com object. Inside "DI API" directory you will find a file "SAPbobsCOM100.dll" (for version 100) and register it with 

%systemroot%\System32\regsvr32.exe C:\DI API 100\SAPbobsCOM100.dll

Note that we use "regsrv32.exe" is used for the 64bit version.

After COM object is installed you can add it from Visual Studio "Add references"

 Once we have the COM object we can start coding. A basic object which is used for connection is called "Company"

  static void Main(string[] args)
        {
            Company oCompany = new Company();
            oCompany.Server = "Sap-10-Sql";
            oCompany.language = BoSuppLangs.ln_English;
            oCompany.DbServerType = BoDataServerTypes.dst_MSSQL2019;
            oCompany.CompanyDB = "MyCompanyDb";
            oCompany.UserName = "admin";
            oCompany.Password = "pass";
            oCompany.DbUserName = "sa";
            oCompany.DbPassword = "pass";
            oCompany.UseTrusted = false;
        
            var lRetCode = oCompany.Connect();
            if (lRetCode == 0)
                Console.WriteLine("Done");
            string sErrMsg = string.Empty;
            oCompany.GetLastError(out lRetCode, out sErrMsg);
            Console.WriteLine(sErrMsg);
            oCompany.Disconnect();

        }

"Server" is the DB server and not an application server.

"CompanyDB" is the name of the database of the database server.

"UseTrusted" controls if DB authentication using DB credential or Windows credentials.

Now if you think everything will work after this, you are wrong :)

Probably you will get the following error

Failed to connect SLD, make sure SLD server is correct and available

This is because you also need to define the URL for SLD server (have no idea what is it, but you need it) and the license server.

You can do it programmatically by setting

oCompany.LicenseServer = "sap-10-app:40000";
oCompany.SLDServer = "sap-10-app:30010";

or you can do it in the conf file. In the same location where "DI API 100" folder is located, search for the folder called "Conf". Inside the conf directory open the file b1-local-machine.xml and modify the following values

Now you are good to go!

Running embedded PowerShell script from .Net program

 I have a web UI that allows writing Powershell scripts. Later this script is being sent to the backend and executed.

In this post, I will focus how to execute the Powershell inside c#. 

First of all you need to add to you project "System.Management.Automation" reference. If you are not sure where to take it from, just open windows PowerShell windows and type

[psobject].assembly.location

The output is the exact location of the DLL.

In my program, the purpose of the PowerShell is to give the user the ability to manipulate the data of some DataTable object. For this purpose the program will person 3 operations.

1. Create DataTable

2. Pass it to Powershell 

3. Change the DataTable from inside the PowerShell

4. Return the DataTable 

Create DataTable

           DataTable dt = new DataTable();
            dt.Columns.Add("ID",typeof(Int32));
            DataRow dr = dt.NewRow();
            dr["ID"] =22;
            dt.Rows.Add(dr);

Create a Powershell object and pass this DataTable to it

           InitialSessionState iss = InitialSessionState.CreateDefault();
            Runspace rs = RunspaceFactory.CreateRunspace(iss);
            rs.Open();
 
            string powershell = @"Param([system.Data.DataTable]$dt)   
                                  [system.Data.DataRow]$row = $dt.NewRow();
                                  $row.ID = 3;       
                                  $dt.Rows.Add($row); 
 
                                  return  ,[system.Data.DataTable]$dt; "; //here is the catch ","
 
 
            using (PowerShell powerShellInstance = PowerShell.Create())
            {
                powerShellInstance.Runspace = rs;
                powerShellInstance.AddScript(powershell);
                powerShellInstance.AddParameter("dt", dt);
 
                DataTable outDt = null;
 
                Collection<PSObject> PSOutput = powerShellInstance.Invoke();

Note the Powershell parameter. It expects to get the DataTable . After this the script adds a new record into the DataTable.

This problematic point that took me some time to figure is how to return the DataTable.

The "," sign before [system.Data.DataTable]$dt; is not a misstake.

If you don't put it the script will return System.Data.DataRow!!! object.

You actually need "," to specify that you are returning the collection of DataRows

What is left now is to read the content of the output. You can check if there are some errors

               //catch the errors
                foreach (var err in powerShellInstance.Streams.Error)
                {
                    var ex = err.Exception?.Message;
                }
and actually read the output DataTable

          //catch the output
                foreach (PSObject pSObject in PSOutput)
                {
                    outDt = (DataTable)pSObject.ImmediateBaseObject;
                }
 
            }


  

If we check the "outDt" is debugger, we will see row with a value "3" that we added in the 

Powershell script

Datatable created by Powershell manipulation


P.S. Point of concern. I am trying to figure out how I can secure the PowerShell 

script from harming my backend. I could not find a way in PowerShell to restrict my 

PowerShell access to my file system. To solve this problem, I wrapped the whole 

code in a custom restricted AppDomain. 
 

Using Nats Streams with .Net

 

The modern approach when you need to get some data from the server to the remote client that that server notifies the client and sends the data to the client without the need from the client to request every second "Do I have something to handle".

The process of receiving the data is called "Subscription".

There are many free and commercial services that provide this functionality out of the box. Today I am going to talk about Nats server. Or more correctly about Nats streams.

Nats server is just Publish/Subscribe server. It means that once you place a message into it, the server will send this message to every client which is subscribed to the server channel. IF some client was down at the point when the server sent the data to the client, the client doesn't get it.

The server also doesn't store the data. It means there is no persistency and no option to recover the data on the server-side.

All the problems from above are solved with Nats Streams. It does has the persistence , it can store the data in memory or file or Db. It does provide an access to historical data.

Now, while Nats Streams is also a server, it is actually a client to Nats server. So it this example I will install both server Nats and Nats Streams. 

The installation will take place on Windows machines.

I will use TLS to enable SSL comunication

To start, download both Nats and Nats Streams 

Nats https://nats.io/download/

Nats Stream https://docs.nats.io/nats-streaming-server/install

As I mentioned I will install on Windows but since Nats is written by using "Go" it can be installed also on Linux or on Docker.

I also will use the certificate I have from "GoDaddy" to set the server for TLS. (You can use also a self-signed certificate as described here).

The problem is that my certificate is pfx file but I need to have a key and certificate file separately.

 I will use openssl (for windows ) to convert the pfx to formats that I need.

Use the following commands

openssl pkcs12 -in [yourfile.pfx] -nocerts -out [yourfile.key]

openssl pkcs12 -in [yourfile.pfx] -clcerts -nokeys -out [yourcertificate.crt]

openssl rsa -in [drlive.key] -out [yourdecryptedkey.key]

Note that you need only "yourcertificate.crt" and "yourdecryptedkey.key"

To start the Nats with TLS support run the following command.

I created to directories "Nats" and "NatsStream" and places each server to the relevant directory. 

Also I placed the "crt" and "key" file to the NatsStream directory

D:\Nats\nats-server.exe -DV --tlsverify --tlscert=D:\NatsStream\yourcertificate.crt --tlskey=D:\NatsStream\yourdecryptedkey.key

Assuming that the server URL is myhost.mydomain.com the server will be started at myhost.mydomain.com:4222 (which is the default port)

To start the NatsStream run the following command. As I mentioned Nats stream is the client for Nats so you need to specify the certificates for the server and for the client

D:\NatsStream\nats-streaming-server.exe -m 8222 -a myhost.mydomain.com --tls_client_cert D:\NatsStream\yourcertificate.crt --tls_client_key D:\NatsStream\yourdecryptedkey.key  --tlscert D:\NatsStream\yourcertificate.crt --tlskey D:\NatsStream\yourdecryptedkey.key -nats_server nats://myhost.mydomain.com:4222

The command is a little bit long and I will explain it.

All the certificate flags are related to the TLS configuration. 

-m 8222 will start some basic monitoring service that is able to provide very basic information about channels and messages which are passed through the server. After the server is started you can access it on HTTP://myhost.mydomain.com:8222.

-nats_server nats://myhost.mydomain.com:4222. As I mentioned Nats Streams is only a client to Nats server, so you need to specify the location of the Nats server.

-a myhost.mydomain.com bind to host address

You can learn more about command-line options here

Now we are ready to start the development.

You will need to packages

Nats client from here and Nats Streams client from here. Those are also available as Nuget packages.

Next, we will write 2 simple programs for the publisher and subscriber.

The programs are very straightforward and based on the examples from the github links above. I am not going to describe every line of code.

I did struggle a bit with SSL so I will focus on the configuration

You do need to load the certificate on the client side. Not the key or crt file, but the original pfx file

Also if you are less concerned with validating the certificate security, you can just always return "true" from the following function. In my case I was interested to see that there is no TLS errors.

After execution of the program you should see the following output (of course you need to modify the server URL and certificate location and password in the code)

Attached the project files here