Automating upgrades with Ansible, fighting with APT auto-starting Cassandra service

Question

I'm preparing to upgrade nodes from 3.11.17 to 4.1.7. I've done the upgrade successfully by hand with some dev nodes, but I am wanting to make this more repeatable with Ansible. I am running on Ubuntu servers, so I have to fight with apt auto starting the service. Wondering what would be the most kosher process of doing these upgrades without apt tossing default files at me and possibly messing stuff up. My thoughts are:

1. drain and stop one node
2. Template in my necessary cassandra.yaml and rack properties files
3. Do the install of cassandra 4.1.7
4. Cope with apt auto starting cassandra

Does this spell disaster? My data isn't in the default path of the config that comes from the package maintainer (/var/lib/cassandra), and some of my config variables are no longer present in 4.1 so I would expect cassandra to crash on startup, seeing the unrecognized variables (which I came across with my dev tests). What I would REALLY like is to install cassandra and give it a good look over before starting, but I am not afforded that luxury. In numerous tests, stopping cassandra immediately after a successful install did not work, cassandra would persist.

Answer 1

This is really more a Debian question that it is Cassandra but I do agree with you that it would be preferable for the service not to be automatically started when the Cassandra package has been upgraded.

You will need to configure Ubuntu to disable the auto-start so you have full control and can manually configure Cassandra with the new settings.

I should add that you can't use the Cassandra 3.11 version of the cassandra.yaml with C* 4.1 as you've already figured out in your testing.

You need to start with fresh copies of C* 4.1 configuration files and configure the new properties with new settings based on the tests you've performed. Note too that the JVM options have been moved to conf/jvm8-server.options and conf/jvm11-server.options in C* 4.0 (CASSANDRA-9608).

Please familiarise yourself with all the necessary upgrade steps between C* 3.11 and 4.0 + 4.1 documented in NEWS.txt. Even though you are upgrading directly to C* 4.1 from 3.11, some of the steps in 4.0 may still apply to your cluster(s). Cheers!