The cloud lets you evaluate the cost of performance optimizations

One of the things I love about cloud computing is you can put an honest price on computing time.  You can than balance the human engineering time required to optimize code (and often have more complex code) vs just paying for the cloud to do it.  The Zillow rent estimate post speaks to this brilliantly:

We implemented the Rent Zestimation process as a software application taking input from Zillow databases and producing an output table with about 100 million rows.

We deploy this software application into a production environment using Amazon Web Services (AWS) cloud.  The total time to complete a run is four hours using four of Amazon EC2 instances of Extra-Large-High-CPU type.  This type of machine costs $1.16/hr.  Thus, it costs us about $19 to produce 100 million Rent Zestimates which is the same as a 3-D movie ticket or about 5 gallons of gasoline in New York City today.

A few things to note about this quote:

  • If your data processing can be done in parallel, you’d have the same cost but at a lower time since you can run all instances in parallel
  • If your data processing must be done in serial, and you want results faster than the largest instance can compute them, buy some coffee you’ll be up doing optimizations.
  • Gas prices are through the roof in New York

Comments

Post a Comment

Popular posts from this blog

Finding CLR exceptions without visual studio

(If you want to understand what exception code 0xe0434352 is, read this post ) Often exceptions are thrown and caught and you don't see them. You probably know how to debug this in Visual Studio, so let me show you how to do it in cdb. Sample Code: class Program { static void Main(string[] args) { foreach (var x in Enumerable.Range(0,2000)) { Thread.Sleep(TimeSpan.FromSeconds(1)); Console.WriteLine("Hello World"); ThrowAndCatchException(); } } private static void ThrowAndCatchException() { try { throw new NotImplementedException(); } catch(Exception) { } } } Output of the application: Hello World Hello World Hello World Nothing about an exception, but you're sure it's happening behind the covers -- fire up cdb: C:\Program Files\Debugging Tools for Windows (x64)>cdb -pn consoleapplication3.exe <SNIP> ModLoad: 000007fe`f7e90000 000007fe`f7eb4000 C:\Windows\Mi...
Read more

Why do I keep getting exception code e0434352?

If you want to know how to debug CLR exceptions using cdb then read this post . Exception code e0434352 is the exception code used internally by the CLR to represent most exceptions(*). Regardless of if you throw a System.NullReferenceException or a System.ArgumentException in C#, you'll throw a SEH exception e0434352 under the covers. A fun way to validate this theory is to watch what happens to the CLR exceptions settings in cdb. Fire up cdb, and see the state of clr exceptions: 0:000> .shell -ci "sx" findstr clr clr - CLR exception - second-chance break - not handled clrn - CLR notification exception - break - handled .shell: Process exited Now, set the exception handler for exception number e0434352 and recheck the value of the clr exception handler: 0:000> sxe e0434352 0:000> .shell -ci "sx" findstr clr clr - CLR exception - break - not handled clrn - CLR notification exception - break - handled .shell: Process exited Armed with ...
Read more

Powershell script to enable windows to capture localhost traffic in wireshark

If you want to understand why the following scripts work read this post . Otherwise just paste the following into an elevated powershell window: Setup windows networking to allow localhost capturing in wireshark: # Find the network configuration that has the default gateway. $defaultAdapter = Get-WMIObject Win32_NetworkAdapterConfiguration | ? {$_.DefaultIPGateway} if (@($defaultAdapter).Length -ne 1) {throw "You don't have 1 default gateway, your network configuration is not supported" } # Route local IP address via the default gateway route add $defaultAdapter.IPAddress[0] $defaultAdapter.DefaultIPGateway Write-Host "Start capturing on localhost by connecting to $($defaultAdapter.IPAddress[0])" Return windows networking to normal configuration: # Find the network configuration that has the default gateway. $defaultAdapter = Get-WMIObject Win32_NetworkAdapterConfiguration | ? {$_.DefaultIPGateway} if (@($defaultAdapter).Length -ne 1) {throw "Y...
Read more