Let's cut straight through the bullshit: if you're an engineer working in a company that deals with sensitive data or operates in a regulated industry, you've probably rolled your eyes at the mention of "compliance" more times than you can count. SOC2, ISO27001, HIPAA, GDPR – these acronyms often feel like speed bumps on the highway of rapid development and deployment.

But here's the kicker: DevOps and compliance aren't natural enemies. In fact, when done right, they're powerful allies that can supercharge your software development process and keep your ass out of regulatory hot water. So buckle up, because we're about to dive into how you can make DevOps and compliance work together without sacrificing your sanity or your velocity.

The Compliance Conundrum in DevOps​

First things first: let's address the elephant in the room. Many engineers view compliance as the antithesis of DevOps practices. DevOps is all about speed, automation, and continuous improvement, right? Meanwhile, compliance feels like a bureaucratic nightmare of checklists, audits, and red tape that is impossible to argue with because someone will just throw a regulatory acronym in your face if you try to.

But here's the truth: The people that developed these frameworks, and regulations didn't do it to annoy engineers and slow them down. They instead have security and the users well-being at heart. And I guarantee that no engineer wants to purposefully builda vulnerable piece of software or leak their users data. So what gives? Somewhere between the ink dropping on those regulatory papers and us building development flows that purpose was somehow lost.

The key to keep the spirit of compliance without slowing down your development is to bake compliance into your DevOps practices from the get-go. This approach, often called "Continuous Compliance" or "DevSecOps," integrates security and compliance checks throughout the entire software development lifecycle. It's not about slowing you down; it's about making sure you have sane practices in place to protect your users and your business. Just like refactoring every now and then to keep your codebase clean and maintainable, compliance, if done right, will make sure your infrastructure and data are safe and secure.

Integrating Compliance into Your DevOps Pipeline​

So, how do we actually make this work? Let's break it down:

1. Version Control: Your Compliance Best Friend​

If you're not using version control, stop reading this article and go set it up right now. Seriously, I'll wait.

Version control isn't just about tracking code changes. It's a compliance goldmine. Every commit is a record of what changed, who changed it, and when. This traceability is music to auditors' ears. Use branching strategies and pull requests to enforce code reviews, which not only improves code quality but also serves as evidence of your change management process.

2. Infrastructure as Code (IaC): Compliance at Scale​

Infrastructure as Code isn't just a cool DevOps practice; it's a compliance superhero. By defining your infrastructure in code, you create a single source of truth for your environment configurations. This makes it infinitely easier to demonstrate compliance with security standards and to quickly remediate issues across your entire infrastructure.

Tools like Terraform or AWS CloudFormation don't just make your life easier; they make your compliance team's job a breeze. And trust me, a happy compliance team means fewer headaches for you.

3. Automated Testing: Your First Line of Defense​

Continuous integration isn't complete without automated testing. But don't stop at unit tests and integration tests. Incorporate security scanning and compliance checks into your CI pipeline. Tools like SonarQube, OWASP Dependency-Check, and custom scripts can catch potential compliance violations before they ever reach production.

Remember: fixing a compliance issue in development is infinitely cheaper and less stressful than explaining to auditors why you have unencrypted PII floating around in prod.

4. Immutable Infrastructure: Reduce Your Attack Surface​

Embracing immutable infrastructure isn't just a trendy DevOps practice; it's a compliance dream. By treating your servers as disposable and frequently replacing them with fresh, up-to-date images, you reduce your attack surface and make it easier to maintain a known-good state.

This approach aligns perfectly with compliance requirements around patch management and system integrity. Plus, it makes rolling back changes a breeze if something does go wrong.

5. Continuous Monitoring: Real-time Compliance​

DevOps teams love their monitoring tools, and guess what? So do compliance folks. Implement robust logging and monitoring across your entire stack. This isn't just about catching performance issues; it's about having real-time visibility into your compliance posture.

Tools like ELK stack (Elasticsearch, Logstash, Kibana) or cloud-native solutions like AWS CloudWatch can be configured to alert on potential compliance violations, from unauthorized access attempts to unusual data access patterns.

6. Automated Documentation: Because No One Likes Writing Docs​

Let's face it: documentation is often the bane of a developer's existence. But it's also crucial for compliance. The solution? Automate it as much as possible.

Use tools that can generate documentation from your code, infrastructure definitions, and CI/CD pipelines. Swagger for API documentation, automated changelog generation, and even AI-powered documentation tools can significantly reduce the manual effort while keeping your docs up-to-date.

7. Access Control and Secrets Management: Lock It Down​

Proper access control is a cornerstone of compliance, but it doesn't have to be a pain in the ass. Implement robust identity and access management (IAM) practices across your entire stack. Use tools like HashiCorp Vault or AWS Secrets Manager to securely store and manage secrets.

And for the love of all that is holy, stop hardcoding credentials in your code or config files. It's 2024, and there's no excuse for that nonsense.

The Human Element: Building a Compliance-Aware Culture​

All the tools and automation in the world won't save you if your team doesn't buy into the importance of compliance. Here's how to build a compliance-aware culture without turning into the fun police:

1. Education is Key: Don't just tell your team to "be compliant." Explain why certain practices are important and how they protect the business (and their jobs).

2. Make it Easy: Provide clear guidelines, templates, and tools that make following compliance requirements as painless as possible.

3. Celebrate Compliance Wins: Did someone catch a potential compliance issue early? Give them a shout-out. Make compliance a source of pride, not just a box to tick.

4. Empower Your Team: Give developers the tools and authority to address compliance issues themselves, rather than relying on a separate compliance team for every little thing.

5. Continuous Improvement: Regularly review and update your compliance processes. If something's not working or is causing unnecessary friction, fix it.

Tools of the Trade: Making Compliance Suck Less​

Alright, let's talk tools. There's a whole ecosystem of solutions out there designed to make DevOps compliance less of a headache. Here are a few worth checking out:

1. Chef InSpec: An open-source framework for testing and auditing your applications and infrastructure.

2. Prisma Cloud (formerly Twistlock): Provides full-stack security and compliance management for cloud-native applications.

3. Anchore: Analyzes container images for vulnerabilities and policy violations.

4. Compliance as Code tools: Solutions like AWS Config Rules or Azure Policy that let you define and enforce compliance rules programmatically.

5. Kviklet: Yeah, I'm gonna plug my own tool here. Kviklet helps you manage secure access to production environments without killing developer productivity. It's perfect for teams that need to maintain compliance while still giving devs the access they need to get shit done.

The Bottom Line: Compliance Doesn't Have to Suck​

Here's the deal: compliance isn't going away. If anything, regulatory requirements are only going to get stricter as data privacy concerns grow and cyber threats evolve. But that doesn't mean we have to choose between being compliant and being agile.

By integrating compliance into your DevOps practices, you're not just checking boxes for auditors. You're building more robust, secure systems. You're reducing the risk of costly data breaches or compliance violations. And most importantly, you're setting up your team for sustainable, long-term success.

Remember, the goal isn't to be compliant for compliance's sake. It's about building trust with your customers, protecting your business, and yes, covering your ass legally. But when done right, it also leads to better software, more efficient processes, and a team that can sleep soundly at night knowing they're not one misconfigured S3 bucket away from a data breach nightmare.

So stop treating compliance like a necessary evil and start seeing it as an integral part of your software development lifecycle. Embrace the tools and practices that make continuous compliance possible. And for fuck's sake, if you're still managing production access with shared SSH keys shared read only accounts to the database,it's truly time to grow up and get serious about security.

DevOps and compliance can be allies, not enemies. It's time we started treating them that way.

Got thoughts on DevOps and compliance? Struggling to make it work in your organization? Feel free to reach out on Threads or LinkedIn. And if you're looking for a tool to help manage secure access to your production environments without sacrificing developer productivity, check out Kviklet. It might just be the missing piece in your DevOps compliance puzzle.

The vulnerability allows an attacker to bypass the authentication process and remotely execute code with root-level access on the system, potentially leading to data theft or system compromise. At risk are OpenSSH versions 8.5p1 (2020) to 9.7p1 (2024) or below 4.4p1 (2006).

The "Remote Code Execution" (RCE) vulnerability has been assigned CVE-2024-6387 and dubbed "regreSSHion" - This is pun to this being a regression bug because it has been reported and fixed in 2006 (CVE-2006-5051) and reintroduced in 2020.

Luckily the vulnerability is (still) challenging to exploit, requiring multiple attempts and timing for a succesful attack - but any serious organization would want to mitigate this risk immediately.
For the real Geeks among us - A basic proof of concept exploit has been published on GitHub. [2]

80% of Companies with Cloud using OpenSSH​

OpenSSH has become the de facto standard for remote access to servers. It is used by DevOps, SysAdmins, Engineers, and Security teams to manage (cloud) infrastructure, and is often the first tool to be installed on a new server.

WIZ, a cloud security company, estimates that 80% of organizations with cloud environments have at least one vulnerable instance of OpenSSH running, putting them at risk. [3]

Qualys has run internet scans and identified 14 million potentially vulnerable OpenSSH servers, which are accessible from the internet(!). [1]

Mitigation 1: Update OpenSSH - or Change Config and Risk DOS​

Quickly apply patches and update OpenSSH to version 9.8p1, released on July 1st 2024. [4]

If you are unable to update, you can mitigate this vulnerability by changing sshd setting "LoginGraceTime" to "0" in the configuration file. This will eliminate the risk of remote code excution through this vulnerability, but it can make the server susceptible to Denial of Service (DOS) attacks - which is probably a lesser evil than the risk of a remote code execution.

Nasreddine Bencherchali, a Security Researcher on X explaining how to mitigate the RCE risk, and instead be susceptiple to DOS attacks. [5]

Mitigation 2: Use Fail2Ban to Mitigate Both CVE-2024-6387 and DOS Attacks​

In order to mitigate against both risks - the RCE and DOS - you can use Fail2Ban, an open-source intrusion prevention software that comes with a default config file for sshd and can automatically alter firewall configuration to ban IPs after a certain number of unsuccessful login attempts. [6][6]

Florian Roth, a Security Researcher on X explaining how to mitigate both RCE and DOS risk using Fail2Ban. [7]

Mitigation 3: Use VPN to Prevent Access to SSH Port 22 from the Internet​

Use a VPN such as WireGuard or the open-source Netbird in front of the SSH service. This will prevent attackers from accessing the SSH port 22 from the internet, reducing the risk of remote exploitation. [6][8]

How to Identify Exploitation Attempts and Systems at Risk​

To reveal exploitation attempts, set up your monitoring systems to detect multiple occurences of "Timeout before authentication" in the sshd server logfiles. [1]

To identify systems at risk, scan your environment for installations of vulnerable versions of the OpenSSH servers.

References​

1. https://blog.qualys.com/vulnerabilities-threat-research/2024/07/01/regresshion-remote-unauthenticated-code-execution-vulnerability-in-openssh-server
2. https://github.com/zgzhang/cve-2024-6387-poc
3. https://www.wiz.io/blog/cve-2024-6387-critical-rce-openssh
4. https://www.openssh.com/releasenotes.html
5. https://x.com/nas_bench/status/1807821951433416877
6. https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-20-04
7. https://x.com/cyb3rops/status/1808089314707784167?s=48&t=tyFllwqVrpzw2rL-cc8b5w
8. https://netbird.io/knowledge-hub/using-ssh-to-secure-remote-access

• New Role and Connection Edit pages, check them out, should be a lot more intuitive than before.
• A download as CSV button for Select queries. So that you can use your favorite Spreadsheet tool to analyze and browse the data. Or process it further with your favorite programming language.
• Notifications with error messages if stuff goes wrong (hopefully you dont see them so often). But using Kviklet should have come a lot more intuitive due to this.
• Generic OIDC provider support (although I still haven't tested all OIDC providers of course). So you can use gitlab, okta, or whatever you want to authenticate your users.
• Slack and Teams Notifications for new Requests and Reviews, so you don't have to ping your teammates manually anymore!
• Support for role-based access-control to specific connections. You can now use AntMatchers to select a subset of your connections and allow specific roles to create requests on them. This is mainly useful if you have multiple teams using the same Kviklet instance.
• A new default role with readonly permissions, so that logging in as a new user doesn't look so empty! You can also edit the permissions of this default role.

If you have wishes for the next release, feel free to open an issue on Github or reach out to me directly. Progress for 0.5 is tracked in this issue.

In today's fast-paced DevOps environment, ensuring that developers have secure yet efficient access to production data is critical. We have written multiple posts on why exactly this is the case, see here and here. Read-only access to production databases is often a good sweetspot to start with for troubleshooting, analytics, and various operational tasks, that in our experience solves 50%+ of all dev access requests. However, this access must be carefully managed to prevent security risks and maintain compliance. In this guide, we'll walk through how to set up read-only access for developers with comprehensive audit logging using Kviklet, so that your SOC2 or ISO 27001 auditor will be happy.

Why Read-Only Access?​

Read-only access allows developers to:

• Troubleshoot issues: Quickly identify and resolve bugs by examining live data.
• Generate reports: Access production data for ad-hoc reporting without compromising data integrity. Of course ideally you have a working BI setup for this, but as explained in a previous blog sometimes this is just not sufficient (yet).
• Feature Development: Understand the data model and relationships to develop new features effectively. Typical questions are: Is this column nullable? What is the data type of this column? What is the distribution of this column?

However, without proper controls, this can lead to security risks, data breaches, or accidental data manipulation. This is where audit logs and access control come into play.

Setting Up Read-Only Access with Kviklet​

Step 1: Deploy Kviklet​

First, deploy Kviklet as a Docker container. Feel free to go to our repository and copy the docker-compose setup. Otherwise here's a quick setup guide:

1. Prepare the Database: Kviklet needs its own database to store metadata about queries, connections, and approvals. Although you can use MySQL, Postgres is recommended. You can find the official image here: https://hub.docker.com/_/postgres.

2. Start the Docker Container:

   docker run \
   -e SPRING_DATASOURCE_PASSWORD=postgres \
   -e SPRING_DATASOURCE_USERNAME=postgres \
   -e SPRING_DATASOURCE_URL=jdbc:postgresql://localhost:5432/kviklet \
   -e INITIAL_USER_EMAIL=admin@example.com \
   -e INITIAL_USER_PASSWORD=someverysecurepassword \
   --network host \ # Use host network for simplicity otherwise kviklet just uses port 80, which you can also expose instead.
   ghcr.io/kviklet/kviklet:main
   

   If you see Tomcat started on port(s): 8080 (http) in the logs, Kviklet is up and running. Don't be confused by the port though, There is an nginx proxy that makes sure Kviklet is accessible on port 80.

Step 2: Create a read-only user on the database​

Kviklet needs a read-only user to access the database. This user should have the necessary permissions to execute read-only queries. Here's how you can create a read-only user in Postgres:

CREATE ROLE readonly_user WITH LOGIN PASSWORD 'password';
GRANT CONNECT ON DATABASE your_database TO readonly_user;
GRANT USAGE ON SCHEMA public TO readonly_user;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO readonly_user;

If you have specific tables that the user should not have access to, you can also revoke the SELECT permission on those tables. If you use MYSQL or Microsoft SQL Server you might want to ask your favorite AI agent for help or look it up in teh documentation of your database.

Step 2: Configure Database Connections​

After deploying Kviklet, log in to the web interface at http://localhost and configure your database connections:

1. Navigate to Settings -> Databases -> Add Connection.
2. Fill in the connection details for your production database. (username, password, host, port etc.)
3. Configure the number of reviews required for running requests. If you simply want to provide every engineer with read-only access, you should set this to 0.

Step 3: Set Up Roles and Permissions​

The default Developer Role allows developers to access any connection and create Requests. If you do not have any other connections set up yet, this is perfectly sufficient for read-only access. However, if you want to restrict access further, and add more connections (read separate database users) you might want to edit the developer role or create a separate readonly Role taht only has access to your new readonly connection.

1. Create a Read-Only Role:
   • Go to Settings -> Roles -> Create Role.
   • Define the permissions for this role, by choosing the ID of you new readonly connection you restrict the access of any user with this role to this specific connection.
2. Assign the Role to Users:
   • Navigate to Settings -> Users.
   • Assign the read-only role to the relevant users.

Step 4: Implement SSO for Authentication​

Unless you want to create users for your developers manually and create passwords for each one, you should enable Single Sign-On (SSO) to streamline authentication and enhance security:

1. Google SSO Setup:
   • Obtain your Google client ID and secret by following Google's instructions.
   • Configure valid redirect URIs: http://[kviklet_host]/api/login/oauth2/code/google in google.
   • Set the following environment variables in your Kviklet container:

     KVIKLET_IDENTITY_PROVIDER_CLIENT_ID=your-client-id
     KVIKLET_IDENTITY_PROVIDER_CLIENT_SECRET=your-client-secret
     KVIKLET_IDENTITY_PROVIDER_TYPE=google
     
2. Or Keycloak SSO Setup:
   • Set up a client in Keycloak and obtain the client ID and secret.
   • Set the following environment variables:

     KVIKLET_IDENTITY_PROVIDER_CLIENT_ID=your-client-id
     KVIKLET_IDENTITY_PROVIDER_CLIENT_SECRET=your-client-secret
     KVIKLET_IDENTITY_PROVIDER_TYPE=keycloak
     KVIKLET_IDENTITY_PROVIDER_ISSUER_URI=http://[host]:[port]/realms/[realm]
     
3. Assign Roles After Login:
   • After users log in via SSO, assign them the appropriate role in Kviklet. Alternatively you can simply edit the default role to give access to your new connection. Then you no longer have to do anything manually.

Step 5: Enable Audit Logging​

Kviklet automatically logs all database interactions, providing comprehensive audit trails:

1. View Audit Logs:
   • Navigate to Audit Logs in the Kviklet interface.
   • Review all executed queries, including who executed them and the reasons provided.

Step 6: Test the Setup​

Before rolling out to the entire team, thoroughly test the setup:

1. Create Test Users: Assign them the read-only role.
2. Run Queries: Have them perform various read-only queries.
3. Review Audit Logs: Verify that all queries are logged correctly.
4. Congrats you now have a self service setup for your developers to access the production database in a secure and compliant way.

Conclusion​

By following these steps, you can set up secure read-only access for your developers while maintaining comprehensive audit logs. This approach not only enhances security and compliance but also empowers your developers to work more efficiently.

Ready to get started? Deploy Kviklet today and take control of your production database access. For more detailed instructions and community support, visit our GitHub repository. If something doesn't work, or you're stuck somewhere or you want a new feature, don't hesitate to open an issue or reach out to us directly. We're here to help you out.

• An integration with Kubernetes Exec, you can now run commands on other pods after they've been approved through kviklet. The idea behind this is that you can audit your script executions through Kviklet as well as database acces. If this works well we will expand on this further with e.g. a web based console some time?
• Support for MS SQL Server Databases. A wish by the community, if you have other databases that you'd like us to support don't hesitate to open an issue!
• The connection settings have gotten an overhaul so you can now edit all details of a connection after you've created it (except the id)
• Requests with multiple results will now be shown in a paginated matter. So you can put all your selects in a single Request (Or mix in updates and deletes if you really want to).
• After running a statement the audit entry will now contain a little bit of metadata for what happened (rows affected/returned), so not everything is gone if you close the tab.
• Configurable max executions per connection, allows configuring connections so that the same request can be executed more than once (1 is still the default though).
• Fixed all timestamp bugs (no more -2 hours ago on comments, we hope...)

This release marks the first release where we don't only focus on database production access but also provide a feature that allows running bash commands on your kubernetes cluster. This feature is a bit of a change in direction for kviklet as DB access has been the main focus point fo Kviklet so far. But conceptually it does make sense that you would want to audit all your production access and not just the database access. We are excited to see how this feature is received and if it will be used by users. We also believe the is a lot of potential to expand into this direction and cover more use cases and different technology in the future if this works well.

The next release will focus on improving the experience around adding roles and permissions. Currently if you do not want to use our default setup of roles you run into a bit of an ugly UI and probably have no idea what to do there. For 0.4 we plan to change that and make this experience easier and way more intuitive. The permission system itself might also need a bit of an overhaul.

If you have any feedback or feature requests, please don't hesitate to reach out to us via a github issue or on our webpage. We are always happy to hear from users and improve the product based on feedback.

Yes, you want to hire smart people as a leader, and you need to trust your engineers for them to be productive. But even if your hiring is perfect and you never hire anyone that's malicious, the greatest minds make mistakes and giving every developer access to production environments will be an issue eventually.
This doesn't have to be leaking a credential or leaving their Macbook open at a Starbucks. This can even be running the DROP TABLE; statement on the production database instead of the development environment. And I am not making these scenarios up, this happens in the real world even to companies like GitLab.

Why do developers need access to production?​

In some organizations I've heard the argument that developers don't need access to production at all. Actually noone would need it. Or a phrase like: Developers should just write the code and then the operations team will deploy it. But in my opinion this is a very outdated way of thinking. In a modern DevOps style "you build it, you run it" culture, devs need access to the production environment. How else are they supposed to do their job of "running it" or feel any sense of ownership? You don't have to ask someone else for they key to your own car do you? If you're not convinced yet, here are 3 simple examples scenarios that you can probably relate too:

1. If production is down, e.g. a service is not reachable after a migration failed
2. A bug has to be investigated, e.g. a user reports that a payment is not being processed
3. Someone from business wants a report that the BI system doesn't cover yet, e.g. some tax report

In all these cases someone has to log in to prod probably connect to the database and run some queries or statements. And usually the best person to solve a problem or help with the adhoc request is the one that actually wrote the code in the first place and knows how the schema is structured and used.

Why giving everyone access to production is a bad idea​

But as I mentioned earlier, giving everyone access to production systems is a bad idea. And it's not just a security risk, it's also a risk to the stability of the system. The main security risks with production access are in order of likelyhood:

• Innocent mistakes: Running a query on the wrong database system, or even the wrong server, or running an inefficient query that locks the database and brings down a part of the system. Developers execute DROP TABLE; on the wrong database all the time or accidentally run their test suite against the production database...
• Accidental leaks: Developers might accidentally leak credentials by sending them via slack or other messengers. This is a common issue, and as soon as the wrong person gets their hand on the credentials the system is compromised. This can also happen by leaving the laptop open at a starbucks or maybe even by simply getting your luggage stolen at an airport.
• Malicious intent: This is the least likely scenario, but it is a possibility and a desastrous one if it happens. If a developer is disgruntled or even just wants to make a quick buck, they could do a lot of damage. This is especially dangerous if the developer has access to the production database, as they could steal sensitive data, delete user data, or hold the company hostage via ransomware. This of course requires write access.

If you start searching for these scenarios you will find a articles for every single one, so I am not making this up. You need to have a sound setup around your production environment, otherwise one of these will eventually happen.

How not to do it​

A common approach is to have an Operations Team handle such topics, or the so called SRE or DevOps team do it. These teams don't do normal software development but are instead responsible for infrastructure. But giving them the burden of handling database access management usually slows the resolution of problems down since information first has to travel from one team to the other and it's impossible for one team to understand the whole system well enough to be operationally responsbile. It's also very stressful work to be the one that has to fix things permanenlty, and it's not very rewarding either. Firefighting will usually kick up your adrenaline and cortisol levels, but it's not a good way to spend all of your working life.

But somehting is even worse in this setup: It splits the ownership of a function of your software. And this goes very much against the core of a modern devops culture (or, more recently platform engineering). Splitting ownership causes a variety of problems, most notably:

• Blame shifting: If something goes wrong, it's the other team's fault
• Lack of understanding: If you don't have to fix it, you don't have to understand it
• Slow feedback loop: If you have to wait for another team to fix it, you can't learn from your mistakes
• Lack of responsibility: If you're not responsible for it, you don't care about it
• Lack of motivation: If you're not responsible for it, you don't get the satisfaction of fixing it

I personally deem the slow feedback loop as the main issue. If a team doesn't have to operate their software they can't learn from their mistakes. And if you can't learn from your mistakes, you can't improve your software engineering practices and skills. Meaning teams will ship code that is potentially slow, hard on the database, causes inconsistencies in the data or even security vulnerabilities without ever being held accountable for it.

The SRE Team/Devops Team solves it has another issue as well, if a whole developer team gets willy nillly access to production it is also still a security risk. If one of the engineers' credentials gets leaked, the whole system is compromised. Or they accidentally install a malicious package, or even worse, they are malicious themselves. This happened at LastPass earlier in 2023:

Incident 2 Summary: The threat actor targeted a senior DevOps engineer by exploiting vulnerable third-party software. The threat actor leveraged the vulnerability to deliver malware, bypass existing controls, and ultimately gain unauthorized access to cloud backups. The data accessed from those backups included system configuration data, API secrets, third-party integration secrets, and encrypted and unencrypted LastPass customer data.

So what's the solution?​

Do you have to decide between a modern, agile workflow and security? Can you not have both?
To a degree, it is in fact often a tradeoff that is made: Introducing more processes, to at least achieve compliance and on the flip side deal with slower developement. But seldomly do these processes actually make the system more secure, resulting in a sort of security theater.

Sadly the alternative is a bit expensive, but it usually makes a lot of sense to invest into more sensible processes around developer access, so hear me out: It involves building up an internal toolstack that allows engineers to safely access production directly. This is in fact what larger companies often end up doing.
I talked to Engineers at AWS and Azure, and both have fully fleshed-out internal applications that allow engineers to shadow each other's sessions when they access production resources. Allowing to pull the plug on each other in case someone attempts something shady. Aslong as developers work together they have rather free reign on what to do in prod but due to having 2 or more people involved in every action, the risk of something going wrong is drastically reduced.

Core Features of a solution like that should be:

• SSO login: So no credentials are shared, every modern cloud native tool has this
• Extensive Audit trailing: So at least if something still goes wrong you have a trace
• 4-Eyes Principle: This is by far the best measure to actually prevent mistakes*and malicious attacks.

But this sounds quite generic, why is there no prebuilt solution out there for this task?
Well there is a few, Most notably Teleport which at least in the enterprise version could tick all the boxes. But it is also incredibly expensive as well as complex to set up.

Four Eyes Principle​

I want to expand a bit on The 4-eyes principle since it is a very powerful tool in security. It is a principle that requires that at least two people approve an action before it can be executed. It makes it much harder for a single person to do something malicious. It is also a very powerful tool in preventing mistakes, as it makes it much more likely that someone will catch a mistake before it is executed.

We use the Principle is software engineering all the time, code reviews are a form of the 4-eyes principle. But it is also used in other fields, e.g. in the military, where two people have to turn a key at the same time to launch a missile. It is also used in banking, where two people have to approve a transaction above a certain amount. So why not use it in production access?

You can even expand on the 4-eyes principle, e.g. by requiring that the two people have to be from different teams, or that they have to be from different departments. This makes it even harder for a single person to do something malicious, as they would have to convince someone from another team to help them. Or you require CTO approval, or approval of 3 people, or 4 people, or 5 people. The possibilities are endless, each step making it harder for a single person to do something malicious, without losing the agility of individuals having direct access to production.

The best part is that you do not end up with a weak link in your system, as you would with a single person having access to production. If one person's credentials get leaked, the system is still secure, as they would need another person to approve their actions. And if one person makes a mistake, the other person can catch it before it is executed.

Of course I would for a startup usually suggest to give every engineer read access and just require a second person for writes, but the principle can be expanded to any level of security that you actually need.

So if you're just as baffled as we are at why a tool implementing this principle doesn't yet exist yet in the world and we are left with suboptimal or even insecure practices, feel free to have a look at Kviklet which is my attempt at solving this problem once and for all.

In the fast-paced world of DevOps, safely accessing production databases is a crucial competency that balances operational efficiency with stringent security measures. This comprehensive guide explores DevOps database access best practices, ensuring your engineering teams can swiftly address issues without compromising on security or system integrity. We'll cover the importance of giving engineers access, the place of migration tools, analytics, and best practices around maintenance and operational tasks. We'll also look at the role of the Four-Eyes Principle in this post.

Why is it important to give engineers access to production databases?​

We already have a post on this topic, Should Engineers Have Production Access? but let's summarize the key points here.

1. Engineers need production access to ensure they can swiftly resolve issues, such as system downtime, bug investigations, or unautomated processes, directly impacting system performance and reliability.
2. Granting access fosters a sense of ownership and responsibility for the system, aligning with the DevOps philosophy of "You build it, you run it," which encourages engineers to be intimately involved with their products throughout the whole lifecycle.
3. However, this access comes with risks, as even the most skilled engineers can make critical mistakes, highlighting the importance of implementing robust security measures and tools that enable safe production access while maintaining system integrity and ensuring sensitive data remains secure.

We will touch on how exactly to implement these security measures in the following sections.

Using Migration Tools for Safe Database Changes​

Migration tools are a critical component of the modern DevOps environment, playing a key role in safe production database access. They allow engineers to make changes to the database schema and data without having to manually write SQL. This is important because it allows for a more controlled and repeatable process. It also allows for better collaboration between engineers and the database team.

Different languages and frameworks have different migration tools. For example, Ruby on Rails uses ActiveRecord Migrations, while Django uses Django Migrations. If you use Flask you can use Alembic. These tools allow you to write code that describes the changes you want to make to the database, and then apply those changes to the database. This is a very useful approach for anything that can be planned and tested in advance. E.g. schema changes, adding new columns, migrating data into a new format, etc.

Running migrations on production should be automated. In fact the same workflow should be true for all environments. E.g. if you write a migration it should automatically run on your dev, staging and production environment or any other ones you might have in between. You can e.g. use kubernetes jobs for this purpose, if your application runs on kubernetes.

A huge benefit of migrations is that they are written as code and managed within your normal code base. This means any changes to the database are versioned and can be reviewed and tested just like any other code change. This guarantess that whatever restrictions you have on your code are also true for your DB access via migrations, and heavily reduces the risk of mistakes, or even malicious changes on this path.

Analytical Tasks​

Analytical tasks, crucial for data-driven decisions, form another challenge for database security in DevOps. They allow developers and Product Managers as well as business analysts to analyze the products data, but simply giving out the root password is ofcourse not an option.

In a mature setup you will usually find a separate interface for these kind of tasks. This is often a data warehouse or a data lake. This is important because it allows you to separate the analytical workloads from the operational workloads. For smaller startups a simple read replica of the production database might be sufficient. But as soon as you have a significant amount of analytical queries you should consider a separate setup.

The data on this interface can be anonymized or aggregated to protect the privacy of your users. This is important because it allows you to give data access to a wider audience without risking the privacy of your users. This is also important for compliance reasons, as you might have to comply with regulations like GDPR. The fact that this is a separate "read-only" database also makes it a lot safer to give access to a bigger part of your organization.

Maintenance and Operational Tasks​

This is the crutch of most setups today and the main reason why developers need to access production databases. If you have a bug in your system, you need to be able to look at the data to understand what went wrong. Same for a performance issue, or if you need to run a report that isn't automated or part of your analytics setup yet. Most developer support tasks need access to the production database systems in one form or another.

So you need to run an ad-hoc query, what now? Most companies we've seen solve this by handing out secrets on a case by case basis to give temporary access. This approach fundamentally goes against the purpose of setting up a devops culture in the first place, which should make Operations and Engineers grow closer together. And It also doesn't solve the problem of the SRE/Devops team being able to do anything they want on the production database. This is a huge security and also a compliance risk.

To solve this issue you need an access control management workflow. Ofcourse as this is the Kviklet blog I highly suggest you try Kviklet out. It's free and Open Source The Four-Eyes Principle is a great way to solve this issue. It means that every action on the production database has to be approved by a second person. This is a great way to prevent mistakes and also malicious actions protecting any sensitive information in your databases. It also alleviates the need for the SRE/Devops team to be on call 24/7, as the developers can now do the work themselves and approve each others requests.

If you have money to drop onto the problem you might also want to consider a tool like Apono, StrongDM or Teleport, which try to solve secure access in general and not just DB access, but also come with a hefty price tag.

Best Practices for Production Database Workflows​

1. Automate as much as possible: Automation is key to ensuring safe production database access. This includes automating migrations, backups, and monitoring. Automation reduces the risk of human error and ensures that best practices are followed consistently.
2. Use the Four-Eyes Principle: As mentioned above, the Four-Eyes Principle is a great way to ensure that all actions on the production database are approved by a second person. This is a great way to prevent mistakes and also malicious actions.
3. Reduce access to be time bound: If you have to give out access to the production database, make sure it's time bound. This means that the access is only valid for a certain amount of time. This reduces the risk of someone forgetting to revoke access. So ideally you have a way to give JIT(Just in Time) access to the database in one form or another.
4. Don't use shared accounts: This is a basic security principle. Don't use shared accounts. Every person should have their own account and their own access rights. This makes it easier to track who did what and also reduces the risk of someone doing something they shouldn't.
5. Keep your development workflows in mind: Whatever we suggest here is only useful if it doesn't get in the way of your developers. If you make it too hard to access the production database, your company will suffer from slowed productivity. This is why we suggest using a tool like Kviklet, which makes it easy to access the production database, but also ensures that best practices are followed.

Conclusion​

In conclusion, giving developers access to production databases is a critical part of the job for many devops teams. It's also a topic that can be fraught with risk and giving developers access sounds scary. However, with the right tools and processes in place, it can be done safely and securely. We've covered the importance of giving developers access, the place of migration tools, analytics, and best practices around maintenance and operational tasks. We've also looked at the role of the Four-Eyes Principle in this post. We hope you found this post helpful and that it has given you some ideas for how to improve your own production database workflows.

Ready to enhance your DevOps database access practices? Try Kviklet today for safe production database access and join the community driving the future of database security in DevOps. Star us on GitHub or better yet, give us your feedback and an Issue!

And engineers did not always want to have to approve every individual statement. Especially when debugging a production error, they need break glass kind of access. We already have a temp access mode that lets you execute any amount of statements for 1 hour. But the UI for this is yah well... lacking. We definitely can't compete with datagrip.

So I wanted to experiment... in theory nothing stops us from starting a proxy server and listening in on the postgres network traffic between a client application and the server. This would allow us to add all the cool kviklet features, like an auditlog on SSO level in a very seemless manner almost unnoticable for the actual engineers that had to connect to prod.

This post is about how this works, I found it interesting to poke around on such a low level of the postgres implementation. So maybe you do too.

Postgres runs on a binary frontend/backend protocol. This traffic is potentially SSL encrypted but if we ignore that, listening in on the traffic results in fun stuff to read like:

P�����select name, is_dst from pg_catalog.pg_timezone_names
B�����������

Which is not very useful by itself. As noted above I wanted to figure out which statements the user runs on the DB. However we can parse the traffic into individual messages to make more sense of them.

If we look at the postgres docs we can for example see that 'P' from the frontend stand for a Parse Message. Which is defined as such:

Byte1('P') Identifies the message as a Parse command.

Int32 Length of message contents in bytes, including self.

String The name of the destination prepared statement (an empty string selects the unnamed prepared statement).

String The query string to be parsed.

Int16 The number of parameter data types specified (can be zero). Note that this is not an indication of the number of parameters that might appear in the query string, only the number that the frontend wants to prespecify types for.

Then, for each parameter, there is the following:

Int32 Specifies the object ID of the parameter data type. Placing a zero here is equivalent to leaving the type unspecified.

This however isn't a statement execution by itself which is what I wanted to track in Kviklet. An execution always consists of a Parse Message, followed by a Bind message that binds specific parameters. Only once this has been done an Execute message is sent.

Now the fact that the statement is first send with placeholder params which are later binded is a bit problematic for me. Afterall I want to add the whole statement to the auditlog with all the parameters filled not just "?" placeholders.

So I also need to parse the Bind message:

Byte1('B') Identifies the message as a Bind command.

Int32 Length of message contents in bytes, including self.

String The name of the destination portal (an empty string selects the unnamed portal).

String The name of the source prepared statement (an empty string selects the unnamed prepared statement).

Int16 The number of parameter format codes that follow (denoted C below). This can be zero to indicate that there are no parameters or that the parameters all use the default format (text); or one, in which case the specified format code is applied to all parameters; or it can equal the actual number of parameters.

Int16[C] The parameter format codes. Each must presently be zero (text) or one (binary).

Int16 The number of parameter values that follow (possibly zero). This must match the number of parameters needed by the query.

Next, the following pair of fields appear for each parameter:

Int32 The length of the parameter value, in bytes (this count does not include itself). Can be zero. As a special case, -1 indicates a NULL parameter value. No value bytes follow in the NULL case.

Byten The value of the parameter, in the format indicated by the associated format code. n is the above length.

After the last parameter, the following fields appear:

Int16 The number of result-column format codes that follow (denoted R below). This can be zero to indicate that there are no result columns or that the result columns should all use the default format (text); or one, in which case the specified format code is applied to all result columns (if any); or it can equal the actual number of result columns of the query.

Int16[R] The result-column format codes. Each must presently be zero (text) or one (binary).

Oof this is getting complicated. So we have parameters and for each parameter we can get the Object ID of the type from the Parse message.
Object IDs are what postgres internally uses to refer to rows. The IDs listed here refer to a table in postgres called pg_type.
In my postgres instance this table contains 500 rows of all kinds of different types my database system supports:

Now this is relevant because in the Bind message the docs refer to the params formatter codes. If the code is 1 the format is binary. Meaning I don't get the UTF-8 version of the parameter but instead just a binary value. I need to know what kind of type the parameter is so that I can transform it to a string.

Since my code is kotlin the formatting for that looks like this:

"bool" -> {
    (bytes[0] == 0x01.toByte()).toString()
}
"char" -> {
    bytes[0].toInt().toChar().toString()
}
"name" -> {
    bytes.toHexString()
}
"int8" -> {
    ByteBuffer.wrap(bytes).long.toString()
}

With this decyphering done I can now print a query when I receive the Execute Message. This is fun since I can now see all the introspection and settings commands that datagrip executes:

Now for my purpose that wasn't quite enough. Just proxying the traffic is only useful if the engineer already has the password to production. However we want to avoid this since this often leads to password sharing and thereby access inflation over time.

Instead I want to integrate my proxy with the kviklet user management, which could also be an SSO provider (google or keycloak). For this I had to implement the login process of the postgres protocol.

Postgres supports a miriad different ways of login in, there is the old way of just sending a username and password via plaintext, there is an md5 hashed version and there is newer options like SCRAM which is defined in RFC-7677. Since this is a beta and I want to add SSL later anyways, I simply went with the md5 version, this is also what my locally hosted postgres instance was using so it was easier to reimplement.

The flow is the following:

1. The client sends a startup message containing a user a desired database and a bunch of meta information.
2. The server responds with an Authentication Request that contains a salt.
3. The client hashes the username + password + salt and sends it to the server.
4. If everything is correct the server respondsw with AuthenticationOK + ReadyForQuery messages.

After this flow I can again forward all messages to my original postgres connection. This one I btw manage directly via jdbc so I don't have to implement any of the authentication logic on that side.

The exact flow is a bit more complex but you can look at the details in the code if you'd like to: https://github.com/kviklet/kviklet/blob/main/backend/src/main/kotlin/dev/kviklet/kviklet/proxy/postgres/PostgresProxy.kt

With this implemented I can now generate a random password that is only valid for 1 hour, start the proxy with it and forward all traffic to e.g. the root user. All without having to share the password for the root user and with all statements logged in our auditlog. Pretty cool right?

Note this is all quite experimental and still needs some further work before I'd call it production ready. But feel free to play around with it and let me know if you find any problems in a github issue!

The Problem Space and Market Feedback​

Our initial observation was clear: database access, and production access in general, represents a significant hurdle across the tech industry. The existing solutions, including StrongDM, Teleport, and Apono, are not only complex but also carry a hefty price tag. However, as we delved deeper, we discovered that awareness of this problem varies greatly, and so does the willingness to pay for a solution. Highly regulated industries showed readiness to invest in complex, expensive solutions, whereas less regulated sectors remained largely oblivious to the issue, showing little interest in paying for such tools.

The Reality of Selling an Open Source and Self-Hosted Tool​

Selling an open-source and self-hosted tool itself is much different from selling a typical SaaS solution. Telling someone about your product that they then have to install on their own infrastructure instead of a simple "Login with Google"-button on some website, was a significant hurdle in selling. For a security tool like ours, there was, however, no way around it, hosting your database access on someone else's infrastructure is simply way too risky.

The Security Sector itself​

The security sector, in particular, presents a "cold start" problem. Our tool, designed to be simple and appealing to small companies, didn't resonate with its intended audience. Small companies are often indifferent to security, and larger companies seek solutions with a proven track record—leaving us in a catch-22 situation. We also do not want to build a generic security tool, but a tool that is specifically designed for database access. Because this is where we think the building around the four-eyes principle is most important and presents a significant advantage over other tools.

Compliance sells better than actual security​

Companies that are regulated or try to maintain/achieve specific certification are much more likely to buy a security tool. This is not because they are more secure, but because they have to prove that they are secure. This is a very important distinction to make. As an early engineering-led product we heavily focused on the security aspect of our solution. We could also explain why the implementation of this workflow is secure and companies should use it, but decision makers often didn't care about actual security but what it could gain them on paper.

Open Source: Not the Amplifier We Hoped For​

Making Kviklet open-source did not significantly boost our visibility or adoption. While platforms like Reddit, LinkedIn, and Twitter offered some traction, most leads came through direct networking, cold outreach, or meetups. Nonetheless, the community's engagement did lend us some credibility. I also believe that this is a long-term investment and that the community and visibility will grow over time.

Security has a difficult to prove ROI​

One of the most challenging aspects we've encountered is demonstrating the return on investment (ROI) for security solutions. Especially in a market that often prioritizes immediate, tangible benefits, the preventative and intangible nature of security investments makes it a hard sell. This challenge is compounded in environments where efficiency and cost-effectiveness are paramount. Companies are busy trying to cut costs, reducing head count, and finding immediate efficiencies which Kviklet only provides if your current workflow is inefficient.

What’s Next for Kviklet?​

Despite these challenges, we still believe Kviklet is a great tool and solution. We will continue to develop features that we find cool and useful, dedicating our free time to this passion project. We are also open to contributions and eager to support anyone interested in enhancing Kviklet. Companies now have complete freedom to use Kviklet, and we stand ready to assist with any required support or specific feature development. Who knows maybe in the future there will be other opportunities for us to make Kviklet a success.

But I wonder if this actually makes a lot of sense, or if open source isn't by design the more "communist" approach to software development where everyone can contribute and use a tool for free. Which doesn't cooperate well with the goal of building a company that wants to make money.

There have been some interesting developments in recent times with open source-based companies. In March 2022 Snowflake bought Streamlit for $800M. Streamlit at the time was a very hot data visualization tool used by data analysts to easily build interactive web pages that allowed users to explore their analysis. The project had 15.000 stars at the time of exit, and no relevant revenue that would justify this valuation.

These days that doesn't fly anymore, tech valuations have shrunk and especially if you don't have significant cash flow you're in for a hard time. Multiple projects have struggled with this. I've recently worked in the data/analytics area so I am more aware of events there but I'm sure that this is a cross industry trend.

I want to look at "Open source Companies" and how their decision to go open source has played out.

Airbyte is a data integration platform that is partially open source-based, the initial proposition was to build a tool that can integrate all kind of data from different APIs. The connectors themselves should be truly open source while the orchestration layer was licensed.

Airbyte recently realized that this decision led to competitors using their open source connectors as well. Consequently they changed the licensing on the most important connectors (databases) so that only Airbyte themselves are allowed to offer them as a service. This didn't go great with the community and makes the licensing less transparent, making it unclear which parts of Airbyte remain truly open source.

I also believe that the initial reason for starting open source was to make use of the community. Essentially collecting free work that builds into Airbyte product. No one wants to build hundreds of connectors inhouse; it's simply too costly. I doubt this plan will work very well with connectors that are now no longer truly open source. After all, who is willing to contribute code to a company without compensation?"

Surely Airbyte did not like this headline

Only very recently HashiCorp's Terraform went a similar route. It looks like too many competitors to Terraform's cloud offering have crept up. So they decided to lock down the licensing in an attempt to hinder the competition. Terraform had previously benefited very heavily from being this open and transparent as it has become the de facto standard for infrastructure as code. Pulumi and AWS CloudFormation are the only somewhat serious competitors on that regard, but they are both much smaller or limited (as CloudFormation only works for AWS). But what do you do if everyone uses your tool, but no one is willing to pay for it and even if they do pay they choose another vendor?

In response, Terraform took a desperate measure, altering the very thing that had contributed to their success - they restricted the open source license. The community sentiment is split about this decision; some people don't care, while others are outraged. Competitors of Terraform Cloud, which had previously built upon what HashiCorp and the open source community developed, saw no alternative but to fork the repository and initiate their own project, called OpenToFu. The name isn't well liked by a lot of engineers but time will tell how this project split plays out. Did HashiCorp shoot themselves in the foot? Can the community that made terraform big in the first place also bring them down in the same fashion?

Some sentiment over OpenTofu

There is more examples of license changes in the history of tech (ElasticSearch, anyone?); sometimes they go well, sometimes not. The change itself is rarely taken positively by the public and often hurts the image of a company.

These examples make me wonder, is it even worth it to build a company based on an open-source product? If the community bonus can just as easily turn into a curse, do you really want to be put at their mercy?

Some companies seem to be more successful at managing the open source offerings, the most prominent example is GitLab, I believe. GitLab was started as an open source project and only turned into a company afterwards, but it seems to manage the split rather cleanly so far. GitLab also offers a bunch of licensing options you can either use their cloud Saas hosting (with a typical free option) or you self host, but you can also self host and pay for an enterprise license. Some companies do not want their code to live in the cloud. GitLab is very intentional about what features they put in which version. But I also don't see GitLab benefiting particularly from open source contributions. Sure someone will fix a bug every now and then, but would the company work without the tool being open source? Probably yes.

I think this is a very key takeaway when deciding to go open source or not: Your business needs to work without the help of the open-source community. Yes sure, transparency is great, and if people love your software and want to help then it's amazing if they can do so. But if your product hinges on the free distribution via community building or the 'free work' from open source developers, you are playing with fire.

My co-founders and I are currently building a piece of software and we have decided to go with the open source approach for a part of our software (typical Open Core). But the goal here isn't that we want the community to do the marketing for us or build our product (although any of these effects are of course appreciated if they happen), we are building a security critical tool and security software should be secure by design not by obscurity. In this way open source is our way of fostering trust.

Moreover it will allow companies that don't have the budget to invest into security to start with a sensible safe solution without any investment. We couldn't gain these companies as a customer anyway, but letting them sit with unsafe practices is not a great solution either.

I am personally a big fan of open source as a concept, but I don't like businesses trying to take advantage of the ecosystem and community.

A digital vault as an analogy to a database, treasuring the most valuable assets of an organization. Created with Stable Diffusion

This was our first blog article, originally published on LinkedIn

In today's digital age, database access remains a vulnerable aspect for many organizations. Think of databases as the vaults storing a company's most valuable assets - personal data (PII), proprietary data, crypto keys, and other sensitive information. Yet, the reality is that the way organizations handle database access is far from secure.

Table of content:​

• Creating Awareness for Overlooked Insecurities in Data Practice
  • Table of content:
  • Past Data Breaches Highlight Importance of Secure Data Access
  • Root Cause are Insecure Practices
  • Manual Data Access: More Frequent and Risky Than Assumed
  • Database Access Management: The Current Landscape
  • Balancing Productivity and Security
  • A Call to Action
  • References

Past Data Breaches Highlight Importance of Secure Data Access​

Historically, the compromise of credentials and privileged accounts has been the most prominent reason for notable data breaches. Google researchers found that 41% of compromises in 2022 were to blame on weak passwords, while Mandiant found that credential theft as an entry vector rose from 9% to 14% between 2021 and 2022 [1].

In 2018 the Marriott hotel group detected a data breach that serves as a case in point: Hackers got access to privileged accounts and were able to extract ~500m customer data from the booking database. The breach took place in 2014, but wasn't discovered until 2018 and Marriott subsequently got issued a $24m GDPR fine [2].

But it's not always malicious intent that wreaks havoc: In 2017 GitLab experienced an 18h outage that impacted ~5000 projects worldwide after an engineer executed a command on the wrong database cluster [3]. Or consider the FAA pilot-alerting system outage in 2023 that led to a nationwide ground stop, delaying 34k flights in the US. Engineers accidentally corrupted the live database, forcing the pilot-alerting system to go offline overnight [4].

Root Cause are Insecure Practices​

Throughout our conversations with engineers and CTOs, and our own experience as (cybersecurity-) engineers, we've noticed fundamental flaws in how organizations provide access to data, exposing them to a great risk. We've witnessed immature processes and insecure practices both in formalized processes and day-to-day operations. This is particularly relevant when organizations are solely reliant on the basic tooling that many (SQL-)databases offer, underscoring the urgent need for rectification. Below are the most common flaws, that we've observed:

1. Authentication Lapses: The absence of Single Sign-On or SAML means users need multiple credentials
2. Shared Accounts: Often, multiple users operate under a single account, making it difficult to trace actions back to individuals
3. Missing Credential Rotation: Credentials that aren't periodically rotated, increase the window of opportunity for malicious actors to exploit potentially compromised, leaked, or outdated credentials
4. Neglected Account Decommissioning: When account decommissioning is treated as an afterthought (e.g., when an employee is leaving the company), it results in overlooked shadow and unmanaged accounts, which can be exploited by hackers
5. Process Circumvention: Some users might bypass standard processes (like using JIRA for access requests) in favor of shortcuts, leading to unmonitored activities
6. Improper Audit Logs: Without correctly implemented audit logs and monitoring, tracing database activities become challenging

Compounding the risks is the absence of rigorous QA in database operations. Typical Software Development Life Cycles (SDLC) involves many steps that can prevent errors from reaching production, including CI/CD pipelines, code tests, and code reviews. However, databases, once granted access, can be freely manipulated in live environments.

Manual Data Access: More Frequent and Risky Than Assumed​

As we deep dive into the challenges associated with data access, we've pinpointed the most common scenarios for data access.

We're intentionally excluding BI/data science and programmatic data access scenarios from our discussion. While these domains present distinct security challenges, they benefit from more mature processes and tooling:

• BI and data science: Frequently facilitated through data lakes, which store a copied subset of the live data, often anonymized or pseudonymized to ensure data privacy. Data lakes are ususally interacted with using analytics tools, such as PowerBI, Tableau, and Python libraries
• Programmatic data access: Refers to the systematic ways software applications interact with databases, incl. cloud databases and SQL. Key methodologies include using dedicated APIs or SQL queries. Authentication frequently uses username/password, API tokens or client certificates

What we want to highlight are manual database access scenarios, as we think that the existing procedures around authentication and authorization are fundamentally flawed. Below are the most common situations in which manual database access occur:

• Incident resolution and 3rd level-support: The ideal scenario involves engineers using specialized tools to troubleshoot and resolve issues. However, what we see is that tools are often unavailable or insufficient, especially for complex incidents. In such cases, engineers require direct production access to diagnose and fix bugs, or probe into issues reported by users
• Feature Development: While developing new features, engineers often need to understand the current data distribution (e.g., for performance evaluations) or validate assumptions about the data. To facilitate this, engineers are either provided access to a replica database or granted direct access to the live database
• Ad-hoc access for reporting: Business analysts sometimes require direct access to production databases, when the data in the BI data lakes is insufficient, not up-to-date, or changed in a way that it corrupted reporting dashboards

In short, manual database access occurs a lot more frequently than many realize and carries most of the problems we highlighted before. Let's take a closer look at how organizations manage and orchestrate manual database access.

Database Access Management: The Current Landscape​

Through our research, we've identified a spectrum of approaches to managing production database access, ranging from early stage approaches to those adopted by mature organizations:

1. No controls: The norm in young and unregulated companies, where access is freely granted upon request and credentials are often provided unencrypted and shared amongst employees
2. Access review process in place: While there are ticketing systems like JIRA workflows in place to review, approve, and document access requests, they often lack strict enforcement. Moreover, procedures of account decommissioning and credential rotation are commonly neglected and treated as a secondary concern
3. Ops team as intermediaries: Some companies deploy dedicated ops teams to serve as intermediaries. They ususally create and provide access to data replications/ data lakes that add to storage expenses, or handle data requests firsthand. This reliance on the ops team creates a bottleneck, preventing engineers from engaging in devops practices. Furthermore, due to their elevated access privileges, these team members can become prominent targets for cyberattacks
4. Internal ad-hoc solutions: While developed in-house and tailored to the organization's needs, these solutions can consume significant developer resources for development and maintenance. We've come across a range of inventive methods, from GitOps workflows to using AWS Lambdas for granting access. Although these methods effectively address specific security challenges, they might not cover all aspects comprehensively
5. Commercial software solutions: In 2023, the market for commercial access management and data security solutions is estimated to be worth $25bn. These solutions typically offer features like RBAC, SSO, dynamic secret generation, and auditing [5]. However, they can be challenging to integrate, lack flexibility in customization, and often carry a hefty price tag
6. Holistic custom platforms: We've seeen mature (tech) companies developing comprehensive platforms for database access with features like time-limited access and column/row restrictions. Additionally, some have implemented review and approval mechanisms on query level, upholding the "four-eyes principle" for enhanced security

In conclusion, while the landscape of database access management is diverse and ever-changing, commercial solutions haven't provided the needed balance between security and developer experience/ productivity. Given their complexity and pricing, it's no surprise that many small to medium-sized organizations lean towards building in-house solutions and processes.

Balancing Productivity and Security​

The challenge facing businesses as they grow is speed versus risk: Grant extensive production access, thereby embracing the modern ethos of "You build it, you run it", but risk enlarging the attack surface. Conversely, strictly locking down access to the production environment might bolster security but slows down the pace of development and incident resolution. Such stringent measures, though appearing safe, aren't completely secure. High-privilege accounts still exist and pose a risk, and the potential for innocent mistakes remains.

In our observations, it's common for startups to begin with a more trust-based and liberal approach to production access, to allow for agility and rapid development in the early stages. However, as these companies scale and mature, there's a shift towards more stringent access controls, often driven by regulatory requirements.

In hyper-growth startups in particular the growth often outpaces their ability to implement mature processes. As a result, there can be a period where these companies find themselves in a transitional phase --- moving from the unrestricted environment of a startup to a more structured and compliant operational model.

A Call to Action​

Witnessing the shortcomings first hand at previous organizations we are creating Kviklet, developing an open-source solution inspired by the most mature workflows we have seen. We believe that a community-driven tool can bridge current gaps to improve security, while improving developer experience.

If you resonate with these challenges and have insights into how database access should be managed, get in touch with us and let's co-create a tool that benefits the entire DevOps community!

References​

1. Google, "How leaders can reduce risk by shutting down security theater", Sep 2023
2. CSO Online, "Marriott data breach FAQ: How did it happen and what was the impact?", Feb 2020
3. GitLab, "Postmortem of database outage of January 31", Feb 2017
4. Federal Aviation Administration, "FAA NOTAM Statement", Jan 2023
5. Gartner, "Gartner Identifies Three Factors Influencing Growth in Security Spending", Oct 2022
6. https://kviklet.dev