Database hardening for a trading platform

Introduction

Database vulnerabilities are a silent threat in trading platforms. They lurk in unrestricted access controls, posing risks of data breaches, operational disruptions, and loss of client trust. This case study examines how we identified these risks and implemented a structured, practical approach to mitigate them. By integrating tools like Teleport, enforcing strict access controls, and embedding detailed logging mechanisms, we significantly enhanced our security posture and operational resilience.

Problem statement

Every trading platform depends on its database to handle sensitive operations—from storing client funds to managing trade records. Yet, our initial access controls had critical gaps:

Unrestricted access to sensitive data

Developer accounts could access client funding information, exposing the platform to intentional misuse or accidental exposure.

Data manipulation

Developers with write permissions could inadvertently or maliciously alter critical data, risking financial discrepancies.

Data loss

Permissions to execute destructive commands, such as table deletions, left the system vulnerable to catastrophic data loss.

Lack of auditability

Without logging and audit trails, accountability gaps hindered issue resolution and increased operational risks.

Developer accounts refer to those belonging to engineers and DevOps personnel. These accounts, if compromised, could act as vectors for unauthorized access.

Operational needs vs. security risks

While access should be minimized, it is recognized that developers occasionally need to:

• Manipulate data to fulfill client requests (e.g., updating specific records).
• Query data to trace production issues when source code analysis is insufficient.

These activities must be conducted under strict safeguards to prevent "oops" moments, where accidental actions result in catastrophic data loss or manipulation.

Risk assessment

Proposed approach

Addressing these risks required a phased approach. Each step introduced a new layer of security, designed to mitigate specific vulnerabilities.

Role-based access control

Unrestricted developer access was the root cause of several risks. To address this:

• Enforce least-privilege principles: Developers accessed only the data essential to their roles.
• Differentiate access levels:
  • Read-only access: For troubleshooting non-sensitive data.
  • Write permissions: Granted only with explicit, time-limited approval.
• Provide standby databases: Developers used a read-only copy of the production database for debugging.

Network isolation

Open access points created opportunities for unauthorized interactions with the database. To minimize exposure:

• Restricted database access to approved endpoints or IP addresses.
• Mandated VPN usage or secure proxy connections for all database interactions.

Multi-factor authentication

Insufficient authentication measures left accounts vulnerable to compromise. Implementing MFA added an extra layer of security by requiring developers to verify their identities using multiple factors before accessing the database.

Data masking

To further protect sensitive data, even when accessed by authorized personnel, we implemented data masking:

• Selective masking: Sensitive data like client Personally Identifiable Information (PII) or financial details were masked or obfuscated.
• Granular control: Masking rules were applied based on user roles and specific data fields.
• Dynamic masking: Data was masked in real-time during queries, ensuring that sensitive information was never exposed in its raw form.

Database observability and audit logging

Lack of visibility into database interactions hindered accountability. To address this, we:

• Implemented robust logging: Tracked every database interaction, including queries, data changes, and administrative actions.
• Set up alerts: Suspicious activities, such as bulk deletions or schema modifications, triggered instant notifications.
• Made logs tamper-proof: Ensured secure storage to prevent alterations.

Break glass access

In emergencies, developers needed immediate access to resolve critical issues. However, such access carried risks if not carefully managed. We implemented a "break-glass" process:

• Multi-party approval: Emergency access required sign-offs from multiple stakeholders.
• Time-limited access: Permissions expired automatically after a set duration.
• Comprehensive logging: Every action during emergency access was logged for accountability.

Technical implementation

System architecture

We used Teleport as the central platform for managing access controls and monitoring database interactions. The architecture featured:

• Public network: Developers authenticated via HTTPS or CLI (tsh) to obtain access certificates.

• Teleport proxy: Served as the gateway, enforcing MFA, role-based permissions, and secure connections.

• Private network: Hosted the database tier, segregated into read-only and write-only instances, and the logging infrastructure.

• Event aggregator: Used Fluentd to process and route logs to tamper-proof storage and notification systems.

• Notification system: Alerted administrators to suspicious activities and provided actionable insights.

Workflow

1. A developer authenticated via Teleport, receiving a temporary certificate.
2. The Teleport proxy validated their permissions before granting access to the private network.
3. Logs of all interactions were processed by the event aggregator and stored securely.
4. Alerts were sent to the security team for any suspicious activities.

Masking data

We hide some sensitive information in our tables to keep data safe. Most of these fields stay hidden forever. However, a few can be accessed with special permissions when needed. Right now, we use postgresql-anonymizer for data masking and follow this process:

1. Identify the table: Find out which table you need access to.
2. Request the tight role: Use the table name with unmasked_ as the role name.

For example, if you need to see hidden fields in the deposits table, request the unmasked_deposits role.

Request a new role for extensive access

If there is a special request for an action beyond the permissions of the existing role, the requester must follow this protocol to perform the action:

sequenceDiagram
    participant User
    participant Approval as Teleport
    participant Database
    participant Audit as Audit logs

    User->>Approval: Submit emergency access request
    activate Approval
    Note over User,Approval: Includes: reason, duration, resources

    loop Approval Process
        Approval->>Approval: Notify approvers
        Approval->>Approvers: Wait for the request to be approved
    end

    alt Request Approved
        Approval->>Database: Grant temporary permission
        activate Database
        Database->>Approval: Acknowledge credential issuance
        deactivate Approval

        Approval->>User: Notify approval with the new permission grant

        User->>Database: Connect with new permission
        activate Database
        Database->>Audit: Log connection attempt

        Note over Database,Audit: Auto-cleanup after X hours

        loop During Access Period
            User->>Database: Execute queries
            Database->>Audit: Log operations
        end

        alt Time Limit Reached
            Database->>Database: Terminate session
            Database->>Audit: Log session end
        else Manual End
            User->>Database: End session
            Database->>Audit: Log manual end
        end
        deactivate Database

    else Request Denied
        Approval->>User: Notify rejection
        Approval->>Audit: Log rejected request
    end

Workflow summary:

1. Developer initiates the request.
2. Approvers evaluate and approve the request via Teleport.
3. Developer performs required actions with temporary permissions.
4. All activities are logged, and permissions are automatically revoked after expiration.

Results and benefits

The implementation delivered measurable benefits:

• Enhanced security: Reduced risks of unauthorized access, data breaches, and misuse.
• Improved data integrity: Maintained through RBAC and robust logging.
• Operational efficiency: Developers performed essential tasks without compromising security.
• Accountability and traceability: Comprehensive logs enabled rapid issue resolution.
• Increased client trust: Demonstrated commitment to safeguarding sensitive data.

Conclusion

This case study highlights how robust access control measures can transform database security in a trading platform. By layering tools like Teleport, enforcing RBAC, and integrating detailed observability, we not only mitigated immediate risks but also established a secure foundation for future growth. These measures underscore the importance of proactive security in maintaining operational resilience and client trust.