The multi-pods deployment allows for global distribution following regional settings permitting full redundancy across the globe.
Each pod is currently run in the AWS cloud but could be run in any facility.
Multi-pods Architecture
- Pods can be deployed in various regions to provision for full redundancy.
- Regional pods are kept in communication with other region’s pod(s) for global data distribution.
- A single pod can also be deployed across different physical locations to remove single point of failure.
- Regional content remains in the region’s pod(s) so there is no latency distribution cost for that region.
- Intra-pods communication is limited to only the real-time content being requested region to region (cross ocean bandwidth optimization).
MarketDesk POD Federation – following AWS availability zones
Every region has at least two pods
Pod Tiers
A pod consists of 4 tiers implementing the system’s functionality.
Content Acquisition Tier
The content acquisition tier implements all the data input facilities.
Feed handlers (FIX, MKF, JSON, and XML)
FTP servers
Contribution servers (JSON over SSL or secure web socket connections)
Excel RTD handlers (direct contribution via excel RTD)
Middleware Tier
MarketDesk developed its proprietary messaging middleware for internal usage. This middleware is optimized for real-time, high velocity, fully redundant to allow the pod’s services intra communication.
Major functionality includes but is not limited to:
- Physical location abstraction
- Fully redundancy and self-healing. No single point of failure
- Self-describing messaging: content agnostic
- Native master/multiple slaves design for crucial service
- Native real-time load balancing across multi-sources
- Secure communication
- Data access auditing.
Content Distribution Tier
Administration Tier
Databases
Permission servers
Active monitoring systems
Cloud Platform
MarketDesk was built on Amazon Web Service leveraging Cloud native tools that provides Scalability, Security and BCDR. Our main mission is to easily scale and deploy our services in a secured manner to our customers across the globe. Using some of the AWS services helps us just do that.
AWS Cloud Security Hub
Identity and access management, detective controls, infrastructure protection, data protection, compliance
Technology Stack
Backend Systems
All MarketDesk code has been developed and is maintained in New York.
Linux based system on multicores servers.
C++ multithreaded design.
Proprietary real-time distributed system.
Web systems
Apache
(Web server)
HTML5
(WebSocket, CSS)
React
(JS lib)
API
MarketDesk offers a range of protocols to contribute and access data:
- Contribution wire protocol: SSL / JSON messaging protocol
- Feed wire protocol: SSL and WSS connection with FIX, XML, MKF, JSON messaging protocol
- WEB URL API: HTTPS connection, CSV, XML and JSON data format, Python wrapper
