The MarketDesk cloud infrastructure is a distributed multi-pods deployment with each pod following a multi-tier design. The multi-pods deployment allows for a global distribution following regional and allows full redundancy across the globe.

This image has an empty alt attribute; its file name is MarketDesk-Platform-Infographic-V3-01-1024x576.jpg

Each pod is currently run in the AWS cloud but could be run in any facility.

Multi-pods Deployment

  • Pods can be deployed in various regions to provision for full redundancy.
  • Regional pods are kept in communication with other region’s pod for global data distribution.
  • A single pod can also be deployed across different physical locations to remove single point of failure.
  • Regional content remains in the region’s pod(s) so there is no latency distribution cost for that region.
  • Intra-pods communication is limited to only the Real-time content being requested region to region (cross ocean bandwidth optimization).
This image has an empty alt attribute; its file name is MarketDesk-POD-Infographic-1024x576.png

Pod Tiers

A pod consists of 4 tiers implementing the system’s functionality.

Content Acquisition Tier

The content acquisition tier implements all the data input facilities.

  • Feed handlers (FIX, MKF, JSON, and XML).
  • FTP servers.
  • Contribution servers (JSON over SSL or secure web socket connections).
  • Excel RTD handlers (direct contribution via excel RTD).

Middleware Tier

MarketDesk developed its proprietary messaging middleware for internal usage. This middleware is optimized for Real-time, high velocity, fully redundant to allow the pod’s services intra communication.

Major functionality includes but is not limited to:

  • Physical location abstraction.
  • Fully redundancy and self-healing. No single point of failure.
  • Self-describing messaging: content agnostic.
  • Native master/multiple slaves design for crucial service.
  • Native Real-time load balancing across multi-sources.
  • Secure communication.
  • Data access auditing.

Content Distribution Tier

The content distribution tier implements all facilities to massively distribute market data to many remote end points. We utilize horizontal scaling so in a modern cloud, the system scalability is infinite.

  • Real-time feed distribution servers (SSL or WSS JSON, XML, FIX, and MKF format).
  • Web-socket servers.
  • Excel RTD servers.
  • WEB REST API servers.
  • FTP servers.

Administration Tier

This implements all the administration services needed by all other tiers.

  • Databases
  • Permission servers
  • Application file servers and network file server
  • Active monitoring systems
  • Reporting tools and email alerts

Cloud Platform

MarketDesk was built on Amazon Web Service leveraging Cloud native tools that provides Scalability, Security and BCDR.  Our main mission is to easily scale and deploy our services in a secured manner to our customers across the globe. Using some of the AWS services helps us just do that.

  • AWS Cloud Security Hub
  • AWS Guard Duty, Inspector, IAM Access Analyzer
  • Identity and access management, detective controls, infrastructure protection, data protection, compliance
  • ELB, EC2, ELB, EFS, S3, Route 53, Cloud Watch, RDS

Technology Stack

Backend Systems:

All MarketDesk code has been developed and is maintained in New York.

  • Linux based system on multicores servers.
  • C++ multithreaded design.
  • Proprietary real time distributed system.

Web systems:

MarketDesk is a major supporter of open source software. Our core web/mobile application stack is powered by:

  • Apache (Web server)
  • Rivet (Web Application Framework)
  • HTML5 (WebSocket, CSS)
  • React (JS lib)

API

MarketDesk offers a range of protocols to contribute and access data.

  • Contribution wire protocol: SSL / JSON messaging protocol.
  • Feed wire protocol: SSL and WSS connection with FIX, XML, MKF, JSON messaging protocol.
  • WEB URL API: HTTPS connection, CSV, XML and JSON data format, Python wrapper.