Adventures in Amazon and eBay

So as a back-end developer here at Football Fanatics, I was originally brought on to create or redo the "feed" process which ties into the Amazon API. Since then we have added the eBay API interface. I have done all this with a framework already established using WCF/WF and the Enterprise Application Blocks from MS.

Ouch, these have been tough. The framework has remained stable and posts several million requests either to the database or via SOAP everyday, it's the Amazon and eBay API and subsequently the support from Amazon and the third party eBay API we are using. We don't post directly to eBay we use another company called Channel Advisor which post to eBay for us.

First learning lesson, despite all my complaining about Amazon, from a programmatic standpoint the process that Amazon has developed is at least able to handle a load. I say this with a frown because Amazon uses java. Originally being a java developer, I both understand it and appreciate it, although I think Amazon sometimes goes overboard with the granularity of the process at least the system works. Now for the bad part. The Channel Advisor (CA) system for eBay is written in .NET. They are using what appears to be .NET 2.0 which is good, but the real problem comes in with volume. I am constantly returned errors from their database about victim of deadlock. Obviously being a SQL error, I am a firm believer that a customer should never see this. This says some pretty bad things about the design of your system.

Now I say this to say the following, the .NET system process more on demand and more rapidly in most cases. Why is this?

I have some speculation knowing both java and .NET but I am not writing to discuss memory utilization or anything else. In my effort to upgrade to 4.0 and I am going to show some of the solutions I use to solve problem with the way both the Amazon system and the eBay system interact with SOAP requests and individual things that eBay and Amazon don't necessarily document well. Hopefully this will help just one person.

Amazon has an interesting system where by you post the individual micro pieces that make up a "product". For instance you post images in an image feed, prices in a prices feed and quantity in a quantity feed and such, where with eBay you post everything together. So with Amazon you have to post and then go back and check the status of that post. This presents a problem in that you have to match up what you posted with what you store locally and then handle the error issues separately. However, with the CA eBay feed you send the response immediately and get a response immediately. This poses a problem because you can only send a small quantity of updates at a time and it takes a very long time to respond. This has caused issue with WCF timeouts and means that we have to manage long running processes in order to handle the additional time it takes to run a smaller quantity of products through.

One of the first issues I dealt with was timeouts. This is a tough subject in WCF, kinda. It's not tough because MS provide some easy ways to manage it, it can seem tough though because the documentation covering timeouts specifically is not great, so let me address a few. There are several ways to manage WCF timeouts, the first being in-line with your code.


InventoryService client = new InventoryService();
client.Timeout = 300000;


You see in this code that we identify the "client" as the .NET service reference and then we set the Timeout property to a pretty large number. This is defined in the MSDN as Indicates the time an XML Web service client waits for a synchronous XML Web service request to complete (in milliseconds).
However you will also see in the following configuration section of my services the following:


<binding name="ServiceBinding" receiveTimeout="00:30:00" sendTimeout="00:15:00" closeTimeout="00:15:00" openTimeout="00:05:00">
<reliableSession enabled="true" inactivityTimeout="00:30:00" />
<security mode="None" />
</binding>


So pay close attention:

recieveTimeout is defined as: Gets or sets the amount of time a TcpClient will wait to receive data once a read operation is initiated.

sendTimeout is defined as: Gets or sets the amount of time a TcpClient will wait for a send operation to complete successfully.


open and close only have to do with the opening and closing of the bindings.

If you paid close attention you realized that all three are needed. In the case of the first timeout, the inline code timeout, the is a full timeout that services wait for a request to complete, but the receive timeout only applies once the read operation has started where as the send timeout only applies to the specific send operation. Why do all three apply, amazingly it's only true for the CA feed because of the delay it takes for them to process a sent request in. It would appear as though (not confirmed)that when WCF makes a request it sends back a response saying send complete (makes sense - sendTimeout), then stalls, then begins a receive or read, then a response is sent that a read is completed. So what happens during that stall while the other side is processing. This is what I think is covered by the inline Timeout. Of course yes, the send and receive timeouts apply to the TcpClient while the Timeout inline applies to SoapHttpClientProtocol. These two however tie together to make up a request.

WCF/WF

Since my last post I decided to get a better grip on the new functionality in .NET 4.0. I will discuss some of the great new features in future posts, but for now I want to discuss some recent discoveries in the WCF framework I have created.

I recently revisited much of my code in the framework I created a couple of years ago for database and SOA communication. I thought perhaps implementing LINQ in a few places would help further increase my throughput.

I got interesting but expected results. Using the Distinct() method is memory intensive, especially when searching large blocks of data in a generic list. I am using LINQ in just a couple of the possible places because I got a plethora of OutOfMemory Exceptions. After doing some research into the MSIL of the framework, I see why.

The short answer is test, test, test. LINQ is a very valuable feature and will improve efficiency and ease of code writing/reading in many cases. But it has it's limits. In the case of large Generic lists, you are better using the Find or FindAll method unless you are doing pretty simplistic queries in LINQ.

Finally, I noticed that if you utilize the Enumerable<> type for your return type of LINQ queries then use foreach loops, you can greatly improve the speed at which large operations occur.

Adventures in WCF and WF (cont.)

So, I finally had time to get things working, except the Oracle piece. I am really impressed; faster than WSE 3.0 by my calculations and easier to use.

The Architecture

Router
The heart is or starting point was best suited at the router. The custom router allows me to us WF rules in any manner I want to route incoming requests. For instance, I can for internal apps, use a custom header to by-pass security checks and router incoming requests. Then I can use another rule to route requests based on the namespace or request URL. I can take one incoming http request and route it to a tcp request. This is not much different then the sample that comes with the WCF_Card samples, just a little modification for the business need.

Here is a sample of the core code where we override the core ProcessMessage method. Here we check the rules engine for the forwarding address, then call the base interface to continue processing the message:

void ISimplexDatagramRouter.ProcessMessage(Message message)
{
    EndpointAddress to = this.extension.RoutingTable.SelectDestination(message);
    if (to == null)
    {
        message.Close();
        return;
    }

    // If the router doesn't already have a one-way datagram channel to the 'to' EPR or if that channel is no longer opened, create one.

    ISimplexDatagramRouter forwardingChannel;

    if (!this.extension.SimplexDatagramChannels.TryGetValue(to, out forwardingChannel) ((IClientChannel)forwardingChannel).State != CommunicationState.Opened)
    {
        lock (this.extension.SimplexDatagramChannels)
        {
            if (!this.extension.SimplexDatagramChannels.TryGetValue(to, out forwardingChannel) ((IClientChannel)forwardingChannel).State != CommunicationState.Opened)
            {
                forwardingChannel = new ChannelFactory(this.extension.Bindings[to.Uri.Scheme], to).CreateChannel();
                this.extension.SimplexDatagramChannels[to] = forwardingChannel;
             }
        }
    }

    Console.WriteLine("Forwarding message " + message.Headers.Action + "...");
    forwardingChannel.ProcessMessage(message);
}

Once you have this piece done, all the rules begin to make sense. So from a rules engine standpoint. I really just used the External Ruleset interface built in the samples. Again modified for the business need then recompiled for external use. Using the rules engine and this router (ask if you want the rest of the code) you see you have a very powerful utility for routing almost any type of request.

Next time I will go over the base data service which uses the latest version of the EAB.

Adventures in WCF and WF

So I set off to rewrite my previously written WSE 3.0 SOA in WCF using WF. The concept based on the original plan was to use WCF obviously to replace the services and use WF rules engine to replace what I called actions. The actions were based on work that Avanade had done here where an action defines the settings that are used for connection information or router forwarding information (an action an object in the framework should take). These definitions were stored in a configuration file.

Well, I got off to a rocky start because I basically took the WCF sample with the router and the Echo Service/Calculator service and changed them to match my needs. First lesson, use the WCF configuration tool. The settings in the configuration files for the services were way off after my rewrite. The configuration tool validated and helped me get them back in sync. The second piece is use the "svcutil" command line tool. Man, this is so undocumented and should be more highlighted somewhere, but was a crucial piece of getting the configs with the right settings.

Couple of comments and I will get into more detail later. Microsoft has finally addressed some of my previous huge concerns for security and compression in the WCF framework. I absolutely think what they did in these two areas in WCF are worlds better than any previous attempts. Good work team!

Next, if you actually read this blog, MS has once again fallen short on documentation for some of the advanced stuff. First, LINQ can not be used in a SOA architecture because some of the core classes in the LINQ framework are not serializable. I know there some folks working on this, so hold your breath. LOL. Also, There is no clear documentation on WF specifically addressing the validation of the rules for framework classes such as the OracleClient. I am still working on figuring this one out so hopefully I can pass along some information for others soon. As I can't find anything via a search engine, I am hoping MS will pull through for me soon, because I can't be the only one on the planet with this issue.