A modern trading platform and a virtual trailing-stop engine

The setup

I joined Zecco around 2008, after they had already raised funding, right at the initiation of a major architectural pivot.

The platform I walked into was a two-tier web application, an ASP.NET web front end talking directly to a monolithic Microsoft SQL Server backend. Source control and build ran on the Microsoft toolchain and Team Foundation Server, TFS for short.

The org was sizable for the era: roughly 60 to 80 people in engineering, more than a dozen in QA, plus admins and a substantial product-management organization. Four “architects” split engineering architecture between them, each owning a slice of the application stack. Tammie Kong, my old Bank of America colleague, was one of those four, and she submitted my name and resume for a newly created Chief Architect role, which I landed.

Zecco’s mandate was to move off the two-tier model to a scalable three-tier architecture. That meant pulling business logic out of the front-end app tier into a dedicated services tier, and from there building out mobile and desktop trading applications. I came in at the very start of that transition to lead its architecture.

There was also a small outsourced development team in China. I don’t recall exactly what they owned at that point; their role changed later in my tenure.

Two rounds of layoffs happened while I was there. The first cut roughly 50% of the organization, and the second went deeper still.

My role

Chief Architect, brought in on Tammie Kong’s referral to lead the pivot from a two-tier web app to a scalable three-tier services architecture, and to set the foundation for mobile and desktop trading apps. In practice the role sat above the four existing slice-owning architects, and the first real work was less about technology than about turning four rival fiefdoms into one engineering organization that worked from shared objectives. The story below covers it.

What I built

Zecco was the first time I was inside a large, funded engineering organization with the latitude to look across every aspect of how software was developed and deployed, and to actually fix what I found. That freedom shaped how I worked for the rest of my career.

When I reviewed the engineering and architectural state, the headline problem wasn’t the code, or even the two-tier architecture. It was the system around the software: how work was conveyed from Product Management to the developers, where and when the architects got pulled in, and the artifacts managed in TFS. So instead of starting with the stack, I started with Product Management and TFS.

Next I redefined what an architect is for. As I worked out the architects’ role, I set the measure of success plainly, and it became a core tenet I’ve carried ever since: an architect succeeds to the degree their engineers successfully adopt the architecture, and to the degree that architecture delivers against functional, technical, and operational requirements and goals. The operational dimension was mine to add; most teams at the time only wrote functional and technical requirements, and operability was an afterthought.

From that, a concrete directive. I did not want decision documents full of deep justification. I wanted the decision made, a short guide on how to adopt it, and above all a template in repo form for each of our common project types: web app, middleware services, infrastructure services, and database. Each template translated the standards and practices into actual, ready-to-adopt implementation. Standards you can clone and run, not standards you have to read and interpret. And before any broader rollout, I directed each architect to use their most senior engineers to help fill those templates in. That got the templates right and worked as a live check against the architects’ own documentation and decisions.

The first proof was logging, done right. They began executing that directive on the logging work from the earlier meeting. I intervened on one technical point that mattered, pushing for asynchronous logging with buffered writes rather than the then-default synchronous log writes, which block the application on every write. A good learning in its own right.

The development operating model, PM → TFS → delivery

The next moves mattered most, and they became the backbone of how I’ve run software delivery ever since: a Product-Management model, its manifestation in TFS, and the development process that ran on top of it.

After studying how the teams worked in sprints, how they interacted, and the political friction between them, I gave Product Management one deceptively simple rule: never reject an idea. Every request got recorded in the product backlog with just enough to be useful: who requested it, what the feature was, and only as much detail as we needed to assess effort, benefit, and priority. Nothing died in someone’s inbox, and nothing got polished before it had earned the attention.

Work followed a strict hierarchy, from idea to test. I introduced a sprint backlog that hung off the product backlog. A product backlog item became a parent and a sprint backlog item its child, feature-aligned where that made sense. The breakdown below it was consistent:

• Sprint backlog items broke into user stories.
• Each user story broke into defined tests.
• A hard rule: nobody could file a bug without a parent test. If there was a bug, there had to be a test that should have caught it, which quietly forced coverage to grow with every defect.
• Each sprint backlog item also broke into development sprint tasks, peers to the stories/flows and cross-linked to them.

Delivery became measurable. Every sprint task carried three fields, time estimated, time spent, estimated time remaining, and I built a rollup system so they aggregated up to the sprint backlog item. Each task went to an individual developer, and the rollups drove a sprint burndown with drill-down by org, team, and developer. That gave us near-real-time delivery tracking instead of a status meeting.

Then came the nightly build and the delivery report. We built a build/CI flow that rebuilt the whole stack on demand and every night, pushing the full build to QA. Supporting it, I built a way to clone one or more production databases into QA with anonymization, and a way to rebuild a database from a known base state. QA had an option to flag that the next build should reset the QA database to base before applying upgrades. After each build an automated delivery report went out in three sections, each owned by a different leader:

1. Code commits by engineer, reviewed by the engineering managers.
2. Bugs reported / bugs fixed, reviewed, automated, and verified by the QA manager and team.
3. Product Management: the burndown, plus user stories flagged ready for PM review and sign-off.

For the rest of my time at Zecco the build ran overnight, and that report became how every manager in engineering, QA, and product started their day. They read the report, adjusted or redirected their people, then worked the next sprint’s artifacts. That work started from a product-backlog review, selection, and population, so when we rolled into the next sprint we already had full estimates and a delivery plan.

Releases were planned, not continuous. I deliberately rejected the deploy-many-times-a-day model the consumer-web crowd was evangelizing at the time. We were a trading platform; we could and should take proper, planned releases outside trading hours. Sprints ran as a development period followed by a bug-fix / release-prep period, either 6 weeks, split 4 dev and 2 QA/release prep, or 4 weeks, split 3 dev and 1 QA/release. QA owned workflow test automation with dedicated test engineers, while developers owned unit and integration-test automation. We codified and automated all of it in TFS/VSS with custom build and deploy scripts.

The pyramid approach, the seed of the tank-tread model

We were building three client apps in parallel for web, mobile, and desktop on a shared backend and services tier, and I introduced what I called the pyramid approach to sequencing the work. It is the model that became the tank-tread model, and we still use it today.

Picture a feature as a box with three horizontal bands: front end on top, middleware in the middle, data/backend/infrastructure on the bottom. The width of each band is the effort going into that tier in a given sprint:

• Early sprints form a pyramid. Most effort sits at the bottom, where data, backend, and infrastructure make the widest band; middleware gets balanced, moderate effort; the front end gets just enough to exercise the stack, a narrow top. A classic three-band pyramid.
• Middle sprints are balanced. Effort evens out; all three tiers get roughly equal width as the feature fills in.
• Late sprints form an inverted pyramid. The heavy lift moves to the front end, integrating the full feature set across the three client apps; middleware evolves only as needed; the data/infrastructure layer, already solid, needs minimal attention.

The governing directive was data first. An architect’s very first job on a feature was to understand the data to be stored, its volume and its read/write activity, before anything else. For a concrete example, the team had committed to SQL Server for log storage, so my direction was to index on the write timestamp formatted as yyyy-MM-dd HH:mm:ss.fff so it sorted correctly as text. This predated everyone defaulting to ISO 8601. I also directed the team to keep only a single primary index to minimize the write penalty on a write-heavy, essentially time-series table. It was the best we could do within the platform’s limits, and it worked.

The side effect was the whole point. Because we built and hardened data and services first, the backend and middleware matured early and stayed stable. There was a well-modeled SOAP-style API, and behind it a .NET object-relational data-access layer. That layer was a structured handler that translated to and from the database tables. It returned fully instantiated, populated objects, and collections of them, for use upstream and accepted them back for writes, with a version number carried on each object for optimistic concurrency. DTOs, data transfer objects, shipped as shared libraries to move data across tiers. So each of the three front-end teams built against mature, stable, failure-tested APIs, and each client app matured at its own pace without destabilizing the others.

The result was development that moved forward like a tank. The lower layers of data, infrastructure, and middleware lay the track, and the whole feature rolls forward over it, tier by tier. That image is exactly why it became the tank-tread model we still run today.

Signature technical work

Three pieces from this period stand out.

The first was cross-site failover with no re-login. A requirement was to fail over between the live site and a backup site while losing only non-trading functionality. I built a cross-web-server, in-memory resilient cache mirrored to the second site. The backup site carried full trading capability but not the community/social features or new-user sign-up. Because session state was mirrored, users failed over between sites without having to log back in. It tested and executed flawlessly.

Second came a custom read/write-through caching layer between the data and services tiers. I keyed it on the retrieved data and its version, and it supported write-through and read-to-cache. It had a profound impact on performance for the most frequent lookups, current trades on commonly viewed tickers.

The third piece, virtual trailing stops, was some of the hardest and most interesting work. Alongside stock trading, margin shorting, and a range of options trading, we offered a distinctive virtual trailing stop that tracked a stock’s movement by volume and momentum, letting the stop ratchet up behind a rising stock while resisting being stopped out by short, brief whipsaws.

Operating philosophy, developed and formalized here

Zecco is where I developed and formalized the operating model I still use, and where it proved its value.

Product management as a cycle

Good product management is cyclical, and it intakes from everywhere before it emits. It solicits vision from executives and demand signal from sales, with the caveat that salespeople tend to make excuses when they miss, so weight it accordingly. Market research supplies competitive analysis and total addressable market, TAM. Customer support/services supply real-world feedback, UI/UX supplies mockups, and an information architect supplies terminology alignment. From PM’s requirements and features, the architects craft the architectural documentation before handoff to developer-task crafting by engineering and QA. That documentation explicitly includes operational requirements, meaning performance, operational cost, reliability, compliance, and security, alongside the technical direction: which technologies and which patterns. In that era the Gang of Four patterns and their successors were how we cut cognitive load and translation error. Ownership and accountability run through a portfolio → product line → product structure, and the product-management organization owns adoption and profitability.

The information architect

I came to see this role as essential. The information architect owns product/feature terminology and also service and resource naming, logging conventions, and more, so logs, usage, and cost can be correlated across the system for troubleshooting, analytics, and cost attribution. Naming is infrastructure.

The engineering-org pyramid: an architect is not a senior engineer

I hold a deliberately unfashionable view. A good org does not want a large pool of senior engineers, and an architect is not merely the most senior one. The roles differ in kind, and I structure the org as a pyramid:

• Architects, few of them, build the blueprints rather than hammer nails. Their measure is adoption and delivery, not lines of code. If I find an architect writing production code, they get demoted.
• Senior engineers form a slightly larger, portable pool, pointed at the thorny, genuinely novel problems. Their gift is inventing novel approaches to unique challenges, which is also their trap: ask them to manufacture routine code and they turn every task into a novel challenge and over-invest in it.
• Developers manufacture code with some latitude, pointed at new areas and needs.
• Programmers manufacture code to established patterns and practices; anything outside that scope escalates to developers, then engineers if needed.

Matching the person to the work this way is how a small team out-delivers a big one. That is exactly what happened at Zecco after the second layoff, and the same role-shaped thinking is behind how I run AI-assisted development today. I hold the opposite pattern in equal contempt wherever I meet it: a code org where everyone is an expensive senior-plus “full-stack” engineer, little or no automated testing at the integration and QA layers, product management that never touches TAM or operating cost, and “architecture” that produces no durable documents.

Net result

The platform, internal codenames Arcadia / Hermes / Maia, was a combined social-networking and stock-trading system serving 400K+ users, built for low latency and high availability: .NET / WCF / ASP.NET, Microsoft SQL Server 2005/2008, multi-threaded services and load balancing, early Amazon cloud plus VMware, and TFS 2008 CI/CD across 50+ projects. I led the 5-person architecture team, the four slice-owning architects plus me, through the move toward the three-tier services architecture and the delivery operating model above.

We launched the new three-tier site roughly 6 to 9 months after I joined, as a true commercial-grade product. It had separate data stores aligned to use, with logging in its own database and social segmented from customer data segmented from trading. An integrated online identity-verification system met KYC requirements and produced one of our claims to fame: a brand-new visitor signed up and trading in as little as 20 minutes. The proof that the model worked came later. After the second, deeper layoff, a far smaller team delivered more feature throughput than the larger pre-layoff organization had. The platform was designed to support up to $100 billion in assets under management.

In that same later period, production hosting moved to our clearing firm’s data centers, a primary plus a geographic failover site, delivered as virtual hosts in a dedicated VMware cluster managed by their IT.

The story

The logging meeting is how I set the tone. One of my most memorable early days was, of all things, a meeting about application logging. There was open hostility among the four architects and their respective engineering teams, so I asked all four, and whoever they wanted to bring, into a conference room with a dry-erase board. Each architect showed up with a few of their engineers, and the room physically arranged itself into the problem: four clusters in four corners, more or less shouting at each other.

I mostly just listened for a while. Then I called a timeout, got everyone to stop, walked to the board, and asked one question: what is our objective here? Underneath the noise, it was simple. Everyone wanted logging in a common format, through a common library, written to a common location. The moment I wrote that on the board, the energy changed. I kept standing there and asked the room to help me capture the options for achieving that objective, and we started filling the board together instead of defending corners.

From there I led on the substance. I walked through Microsoft’s logging-levels model and its useful separation of operational logging from AAA logging, meaning authentication, authorization, and accounting. Then I pointed out its gaps, no real debug or verbose tiers, and proposed we adopt that model but extend it. I also flagged that logging to SQL Server was a poor fit. The data is essentially time-series, not relational, so if we wanted to keep SQL Server we’d need to deliberately design the tables, indexes, and so on around the actual read/write patterns, or consider a different data store altogether. That framing proved useful.

That meeting was a turning point for me at Zecco. In one session I established that I was the technical authority, that I would not pick a side between the factions, and that the way we were going to work together was collaboration in service of a clearly stated objective. It reset the room from authority-and-turf to problem-and-objective, which is exactly the culture the three-tier pivot was going to require.

The first round of cuts came, aimed at engineering, and the mechanics have stayed with me. Everyone was emailed to report to one of two conference rooms, with people at the door checking names because a few just followed the crowd. One room kept their jobs; the other did not. The company let most of the architecture team go, including people I respected and who had helped bring me in, and kept me to carry the architecture and the pivot forward.

My own position in it was a strange one. I wasn’t sent to either room, because I was the person who had written the script that disabled the departing employees’ accounts, driven off a CSV of names I had deliberately not been given access to. Building the mechanism that executes a layoff you are surviving is about as pointed a lesson in the weight of this work as I have had. It also meant that, coming out of that round, I was effectively the architect, left to carry forward, alone, the development model and operating practices I had already put in place, covered above under What I built.

Launch day came with a shot at E-Trade. Gabe Dalporto, our CMO / chief strategy officer, had come to Zecco from E-Trade. On launch day he rented a fleet of taxis fitted with Zecco-branded boards on top and paid them to spend the day circling E-Trade’s iconic San Francisco building. It was a cheeky move that fit the zero-commission upstart’s personality perfectly. What I remember most, though, isn’t parties or events; it’s just a lot of work at an accelerated pace.

Running a commercial-grade trading-plus-social platform surfaced problems you only learn by living them. Getting ahead of patterns like that was part of maturing the product.

Then came the second layoff, and a steady state that still sets my bar. After roughly 6 to 9 months of running the pattern, the second and more severe round of cuts came. By then I owned process, all architecture, and some development, and we had onboarded a China-based development firm. What remained was lean: one UI/UX designer, three engineering managers, a small handful of US-based engineers, and a growing pool of engineers at the offshore firm. That offshore team was remarkably disciplined, with essentially zero attrition, and because they followed the architecture and templates faithfully they were excellent at manufacturing new code to spec, so the “tank” kept rolling forward reliably.

What emerged was a steady state I still judge other organizations against: the offshore team delivered code; the build deployed to QA overnight; the three-section delivery report landed every morning; the engineering managers, the product manager Tony Leach, and the QA manager each reviewed their section and then worked the next sprint’s artifacts and priorities; and a build/release manager and I deployed to production at the end of each sprint. With a fraction of the pre-layoff headcount we shipped more. It worked shockingly well, proof that the operating model and the pyramid/tank-tread discipline were the real engine, not the raw headcount.

I left Zecco toward the end of 2009, around October to November, for family reasons. The platform and the organization were in a strong, stable steady state by then, with the operating model effectively running itself and a lean team out-delivering the larger one that had preceded it. It was a hard season personally, and those details stay private. Professionally, I left Zecco better than I found it, carrying forward every practice it had let me formalize.

What I learned

Core tenets that started at Zecco and that I carry forward:

• Re-architecture is modeling under fire. Moving a live brokerage off a monolith meant holding the whole system, the one that existed and the one we were building toward, in my head at once, and laying the path between them without stopping trading.
• An architect’s success is measured by how readily their engineers adopt the architecture, not by the depth of the justification in their documents.
• Ship standards as adoptable templates, not documents. A repo you can clone and run beats a spec you have to read and interpret, for every common project type: web app, middleware services, infrastructure services, database.
• Operational requirements deserve equal billing with functional and technical requirements. Most teams only wrote the first two.
• Fix the system around the software first. The problems with the biggest payoff were in the Product-Management-to-developer handoff and the artifact flow in TFS, not in the code itself.
• Validate standards with the most senior engineers before broad rollout. It gets them right and checks the architects’ own decisions.
• Lead from the objective, not from authority, as the logging meeting showed.
• The engineering org is a pyramid of distinct roles, not a pile of senior engineers: architect, senior engineer, developer, programmer. Match the person to the work; most code should be manufactured to a blueprint, not reinvented.
• Product management is a cycle that intakes vision, sales signal, TAM, and support feedback before it emits requirements; architecture then adds operational requirements before any developer task is written.
• Naming is infrastructure. An information architect who owns terminology, service/resource naming, and logging conventions is what makes logs, usage, and cost correlatable.

Key people

• Tammie Kong, BofA colleague from the late 1990s; one of Zecco’s four architects; referred Paul for the Chief Architect role.
• Tony Leach, product manager, early in his career and the most eager to learn and adopt the new process; worked closely with Paul. Grew to VP of Product Management at Zecco. linkedin.com/in/tonyleach
• Gabe Dalporto, CMO / Chief Strategy Officer at Zecco. He came from E-Trade and ran the launch-day taxi stunt circling E-Trade’s SF building.
• Build/release manager, partnered with Paul to deploy to production at the end of each sprint.
• China-based development firm, the offshore engineering team onboarded in the later period; highly disciplined, near-zero attrition, strong at manufacturing code to the architecture/templates.

Technologies used

On arrival: ASP.NET web front end; monolithic Microsoft SQL Server backend; Team Foundation Server and Microsoft source control. Target: three-tier architecture with a separated services/business-logic tier; mobile and desktop trading apps. Services/data: SOAP-style APIs with proper interface modeling; a .NET object-relational data-access layer, an ORM-style structured handler returning populated objects/collections, with per-object version numbers for optimistic concurrency; DTOs as shared libraries across tiers; text-sortable timestamp indexing, yyyy-MM-dd HH:mm:ss.fff, with a single primary index for write-heavy time-series log data. Process/tooling: customized TFS process and work-item templates, since TFS defined each artifact in XML and that let us re-author project-wide and per-artifact templates; repo-form architecture templates per project type; asynchronous, buffered logging extending the Microsoft logging-levels model with ops vs. AAA, plus debug/verbose. Resilience/performance: cross-site, cross-web-server in-memory mirrored cache for live↔backup failover with no re-login; a custom version-keyed read/write-through caching layer between data and services tiers. Product: online identity verification / KYC integration, new visitor to trading in ~20 minutes; virtual trailing stops driven by volume and momentum; segmented data stores for logging / social / customer / trading. Method: the pyramid / tank-tread, data-first build sequencing across web, mobile, and desktop clients.

Business-history sources: TechCrunch for the Nov 2007 Series B and a Feb 2009 model-change piece, VentureBeat for the launch, Financial News for the 2012 TradeKing merger.