The Setup
I took a contract opportunity that called for Microsoft WCF / C# development at scale, structured as contract-to-hire, with relocation if it converted.
Itron’s platform, as I found it. The smart meter used three transceivers, later integrated toward a single “trifecta” chip:
- communications with in-home devices, such as a smart thermostat,
- lateral RF meshing with nearby meters, and
- cellular backhaul: initially, as I recall, a separate ring/collar fitted on selected meters between the meter and the mains to carry traffic up to the network.
Addressing ran on a custom system, likely a third-party component, with built-in tree/mesh addressing, building and assigning addresses based on the mesh topology.
This was the North American OpenWay-era AMI electric smart-meter push circa 2010, purely electrical, as I remember it. Gas/water were elsewhere in Itron.
My Role
Brought in as a senior systems architect / WCF-and-C#-at-scale developer on contract, contract-to-hire. I focused on two areas, and built real contributions in each:
- Meter data collection from the meter services.
- Firmware upgrade distribution, specifically cryptographic key injection into the meter, managed by the vendor with HSM integration.
What I Built
1: Meter data collection
What I found. On the services side, data ingestion was split across three core services: gateway, processing, and control. A mesh-like scheme registered each server hosting each service instance, with a software routing scheme to pass traffic between meshed nodes. My remit was the scaling of message processing.
The problem. The default design wrapped the processing code in an async wrapper bolted onto WCF hosted in IIS. It had a stack of problems: connections held open for the duration of processing, IIS’s own concurrency/scaling limits, and more. It would not scale to a real AMI rollout.
What I built. I’d been deep in CLR via C# and had studied a capability in .NET on Intel that allowed threadless waits, so I decoupled the WCF message-receive side from the processing side entirely. Received messages landed in a semi-durable in-memory queue, passed by a mechanism that avoided cross-process and, as I recall, even cross-thread marshalling of the message, and processing scaled its threads dynamically and efficiently against actual message volume rather than tying up an IIS/WCF thread per in-flight request. The core of it was I/O Completion Ports, or IOCP, and the threadless-wait pattern in the Asynchronous Programming Model, or APM. At the time it was a genuinely novel approach, and it delivered large gains over what IIS’s built-in threading could do for concurrent execution. The fun of it was being so far down into CPU architecture, .NET, and the CLR that I could exploit capabilities I’d never seen anyone else build to.
How we proved it. We took a meter simulator and scaled it to tens of thousands of virtual meters, running on a large multi-core test rig: a 256-core collection tier and a 128-core meter-simulation tier. Critically, we injected noise: the meter-to-network link was not TCP/IP, so real traffic arrived broken, fragmented, or incomplete, and a tunable noise factor let us reproduce that failure mode at volume. The result was a large scaling and performance increase, which mattered: San Diego Gas & Electric, SDG&E, was beginning to deploy its infrastructure and roll out Itron AMI meters.
2: Firmware distribution & HSM cryptographic key injection
Several constraints collided here:
- Tiny device memory. Whatever we built had to fit a very small embedded footprint.
- Secure remote firmware updates. Updates were pushed to the fleet, and each device had to verify, before flashing, both that the image was complete and intact, by hash, and that it was genuinely issued by Itron, by signature. That required an Itron certificate on the device for signature verification, and the ability to replace/rotate that cert.
- Utility-side device management. The adopting utility also wanted certificate-based management of its own devices, with endpoints able to sign/encrypt their messages. So the utility stood up its own CA, and we had to inject the utility-generated certificate from the vendor side before provisioning/distribution, again, with rotation.
Without going deep into the implementation, the device effectively carried Itron, the vendor, and utility CA certs, a CA-replacement cert to rotate the CA itself, and per-device certs, with vendor-side generation and injection managed through an HSM. All of it had to do real PKI inside a memory-starved embedded device.
This work planted a conviction I carried into later security engagements: a firmware image must be both signed and validated-before-flash. A surprising number of appliance and device vendors do one but not the other, and skipping on-device validation opens the door to serious compromise. I found exactly that class of flaw years later, ~2015, when I was asked to dig into a smart-UPS management system.
Net Result
Re-architected message processing from an IIS/WCF-bound async wrapper to an IOCP/threadless-wait design that decoupled receive from processing and scaled threads to volume: a large throughput and performance gain, validated against a tens-of-thousands-of-meters simulation with realistic, non-TCP/IP, lossy noise injection, on the 256-core/128-core rig. Timed to SDG&E’s AMI rollout. Separately, designed the on-device PKI for secure firmware distribution and endpoint management: Itron + utility CA certs, a CA-replacement cert, and per-device certs, with HSM-managed cert injection at provisioning, inside a severely memory-constrained device.
The Story
The contract ran April to June 2010, and I then took the RMS offer that July.
What I Learned
-
Mastery of the lowest layer pays at the highest scale. Knowing CPU architecture, .NET, and the CLR well enough to use IOCP and threadless waits solved a scaling problem the conventional IIS/WCF pattern couldn’t.
-
Firmware must be both signed and validated before flashing. Designing the meter’s PKI taught me how many device vendors do only half of this. Skipping on-device validation is a serious, common vulnerability, one I later found in the wild: ~2015, a smart-UPS management system.
-
Real cryptography fits in tiny devices if you design for it: on-device CA chain, rotation, and per-device identity inside a memory-starved embedded target.
Technologies Used
Microsoft WCF / C# at scale; IIS hosting; IOCP, or I/O Completion Ports, and APM threadless-wait concurrency; a semi-durable in-memory message queue with marshalling-free hand-off; gateway/processing/control services over a custom mesh node-registry, no true service bus, software routing; RF mesh + cellular backhaul metering network; tree/mesh addressing; a meter simulator scaled to tens of thousands of virtual meters with tunable noise injection: non-TCP/IP lossy link; smart-meter firmware distribution with signed images and on-device pre-flash verification; on-device PKI: Itron CA + utility CA + CA-replacement cert + per-device certs, with rotation; HSM-managed cert generation/injection at provisioning; embedded/memory-constrained crypto; 256-core collection / 128-core simulation test rig. Deployment context: SDG&E AMI rollout.