DevOps Stack Exchange
DevOps Stack Exchange

Questions tagged [distributed-systems]

15 questions
16
votes
3 answers

How to avoid "retry storms" in distributed services?

A "retry storm" is caused when clients are configured to retry a set number of times before giving up, a retry policy is necessary because of packets loss will occur in normal operation of a service. Take this example: If for example the services…
Richard Slater
  • 11,747
  • 7
  • 43
  • 82
10
votes
5 answers

What is self hosted version control system?

What is self hosted version control system and how is it different from github? I mean if I use self-hosted VCS then where does my code go? I mean: does it get uploaded to the server, or it is saved on my local machine? If it is uploaded to the…
Qasim Arthuna
  • 101
  • 1
  • 5
7
votes
3 answers

How to ensure equitable distribution of SQS messages in a distributed systems setting?

I have multiple servers, each having a script polling an SQS queue [all polling the same queue]. So, is there any way I can ensure an equitable distribution of messages to all those clients [i.e. my worker servers here]. Like for example, if there…
Dawny33
  • 2,816
  • 3
  • 24
  • 62
4
votes
1 answer

How to route to specific, of multiple datacenters, with a single domain

The data of each customer only exist in one of several global datacenters, that all answer to the same hostname (ex. api.company.com) Only a single domain for all customers globally I want to make sure that all global customers only have to use a…
Silas Hansen
  • 171
  • 4
4
votes
1 answer

Necessity of mirroring git repos for small team considering switching to Azure

Question: Are there any benefits to mirroring our git repos outside of Azure devops for a small (~7) team of developers who all work in the same office? Background Information I'm a developer who also wears the "DevOps" hat when he has time to do…
Scrambo
  • 141
  • 3
2
votes
2 answers

Identifying a benchmark to compare overall performance of VM virtualization vs hardware

Which performance index is most suitable to find out real performance of your VM limited by sharing and quotas? For example, if you have a physical server with 4 cores and 16 GB RAM, its performance limit might be higher than if you have same on AWS…
Ta Mu
  • 6,792
  • 5
  • 43
  • 83
1
vote
2 answers

Automated Docker swarm reinitialization and token distribution

Imagine you want to reset your Docker swarm from time to time. How do you distribute join tokens to those zillions of nodes? How much automation is possible here?
Ta Mu
  • 6,792
  • 5
  • 43
  • 83
1
vote
1 answer

Distrubuted storage PVC on k3s using OpenEBS is stuck in a pending status while provisioning - waiting on external provisioning, not sure why

I'm trying to setup distributed raid 1+0 storage on my k3s cluster running of 5 raspberry pi 4s running armbian (jammy). Currently I'm seeing the event waiting for a volume to be created, either by external provisioner…
Snorghma
  • 21
  • 3
1
vote
1 answer

Proper technologies to create a horizontably scalable index search

I am sorry for the newbye question. Im trying to figure out the best architecture / set of technologies to implement some very specific system: I have several databases indexes (used for searching data elements) that are broken down into small…
Pablo Carneiro Elias
  • 111
  • 2
1
vote
2 answers

Dynamic environments per client, which is the best approach, if any?

I don't know how to explain this problem better, but I'll try to explain it clearly: Where I work our customers are companies, with lots of users We offer a SaaS solution for them to manage stuff For every customer, there are around 50 right now,…
Leonardo Cavalcante
  • 11
  • 2
0
votes
0 answers

Limiting Access to logs on a specific cluster to Grafana Admins

I have 3 cluster (A-B-C), I installed loki on cluster A using the grafana/loki helm chart, and on cluster B and C I have promtail installed pushing logs to cluster A, how can I make logs collected from cluster A only displayable to Grafana…
Abdel
  • 1
0
votes
0 answers

Kubernetes networking somehow breaks ROS2 message passing

I have 2 identical docker containers running on separate computers in which I want to run ROS2 nodes communicating with each other (via a VPN). When I run them as docker containers there is no problem getting them to communicate. However when I try…
clemfandango
  • 1
0
votes
2 answers

How to implement simple distributed configuration system

I have multiple servers running in data centers around the world. Each server has its own configuraiton database, and cli command needs to be executed to update in-memory cache. I want to automate this system. I have following requirements: Single…
Shadow
  • 111
  • 3
0
votes
1 answer

s3: reading parquet files with a subset of columns as argument; does this reduce data transfer?

I have set of large tables stored as parquets on s3. In python, I'm using: pd.read_parquet(...,columns=columns) I'm reading the files directly from s3, without any database engine whatsoever for preprocessing in between. My question is, will the…
user19976975
  • 1
  • 2
-2
votes
2 answers

How do you monitor status and view logs for multiple distributed java applications at a centralized place?

We have multiple java applications running on distributed hosts in our system. As theses applications are processing huge amount of data, it is designed this way. However, it had been a challenge in getting the status of these applications, view…
Radhika
  • 129
  • 1
  • 6