Troubleshooting Cassandra data sources

Logging

The following logs will help to diagnose dSource/VDB issues related to this plugin.

Cassandra dSource/VDB log files

/{toolkit_directory}/Delphix_COMMON_{appliance_id}/plugin/sc:cassandra_{plugin-id}/Cassandra/log_{os_user}/delphixCassandraDebug.log

Cassandra log files: Cassandra can write messages to log files. The files are named using the following format and can be found at location mentioned in cassandra-env.sh file:

Verify CASSANDRA_LOG_DIR variable in {Cassandra_install_path}/resources/cassandra/conf/cassandra-env.sh file

For Example:

/{toolkit_directory}/Delphix_COMMON_{appliance_id}/plugin/sc:cassandra_{plugin-id}/.delphix/{<binary_folder_name>_delphix_<dataset_name>_n_<node_number>}/system.log

To make it easier to watch a Cassandra startup log without knowing the full filename, a symlink with the short filename cassandra.log is also created. This symlink points to the most recent log. The Cassandra startup log location on staging or target host can be found at below location:

{delphix_mount_path}/<dataset_name>/<binary_folder_name>_delphix_<dataset_name>_n_<node_number>.log

Troubleshooting Queries

Identify DataStax Cassandra Version

{Cassandra_install_path}/bin/dse -v
For example: 
/u02/cassandra/node1/bin/dse -v
6.8.25

Identify Open-Source Apache Cassandra Version:

{Cassandra_install_path}/bin/cassandra -v
For example: 
/u01/cassandra/apache-cassandra-4.0.7/bin/cassandra -v
4.0.7

Start DataStax Enterprise Cassandra multi-node cluster:

{Cassandra_install_path}/bin/dse cassandra

Stop DataStax Enterprise Cassandra multi-node cluster:

{Cassandra_install_path}/bin/dse cassandra-stop -p {pid}

Start Open-Source Apache Cassandra multi-node cluster:

{Cassandra_install_path}/bin/cassandra

Stop Open-Source Apache Cassandra multi-node cluster:

# Terminate the active cassandra process using:
kill -9 {pid}

Start interactive Cassandra CQL shell:

{Cassandra_install_path}/bin/cqlsh {host} {port} --username {username} --password {password}"

Start interactive Cassandra CQL shell with SSL encryption:

{Cassandra_install_path}/bin/cqlsh {host} {port} --ssl --username {username} --password {password}"

Configure a cqlshrc file that contains CQL shell session settings that are used when the CQL shell starts.

1. Copy the cqlshrc.sample.ssl file to the ~/.cassandra/ directory. If the Cassandra base installation directory is located at /u02/cassandra/dse-6.8.33, then the sample cqlshrc file will be located under resources/cassandra/conf/cqlshrc.sample.ssl

2. Rename the file to cqlshrc copied in Step 1 under ~/.cassandra/cqlshrc directory.

3. Modify or update the cqlshrc file as per your cluster configuration. For example:

   CODE

   > cat .cassandra/cqlshrc 
   [authentication]
   username = dlpxadmin
   password = *******
   
   [connection]
   hostname = 10.11.12.13
   port = 9042
   ; factory = cqlshlib.ssl.ssl_transport_factory
   
   [ssl]
   certfile = /home/delphix/certs/rootca.crt
   ; optional - true by default.
   validate = true
   userkey = /home/delphix/certs/client_key.key
   usercert = /home/delphix/certs/client_cert.crt_signed

4. Restart the CQL shell.

   CODE

   > {Cassandra_install_path}/bin/cqlsh --ssl
   Connected to datastaxstg at 10.11.12.13:9042.
   [cqlsh 6.8.0 | DSE 6.8.33 | CQL spec 3.4.5 | DSE protocol v2 | TLSv1_2]
   Use HELP for help.
   delphixadmin@cqlsh> 
   

5. Verify Cassandra Cluster Status:

   CODE

   For example:
   /u02/cassandra/node1_delphix_dsestg_n_0/bin/nodetool status
   Datacenter: Cassandra
   =====================
   Status=Up/Down
   |/ State=Normal/Leaving/Joining/Moving/Stopped
   --  Address    Load       Tokens       Owns (effective)  Host ID                               Rack
   UN  127.0.0.1  185.97 KiB  1            61.6%             35a2a647-f3b8-4f2d-ad25-ab8376bc12ad  rack1
   UN  127.0.0.2  148.97 KiB  1            44.9%             5f1b513e-6caf-4faa-bd26-4e9344689c80  rack1
   UN  127.0.0.3  148.86 KiB  1            93.5%             cf4a0f6f-ffe5-40a4-b68d-fec5e449f341  rack1

6. Verify Cassandra Cluster setting across all nodes:

   CODE

   {Cassandra_install_path}/bin/nodetool describecluster
   For example:
   /u02/cassandra/node1_delphix_dsestg_n_0/bin/nodetool describecluster
   Cluster Information:
       Name: dsestg
       Snitch: com.datastax.bdp.snitch.DseSimpleSnitch
       DynamicEndPointSnitch: enabled
       Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
       Schema versions:
           a100bba8-50a7-3c2f-959d-695342f1d061: [127.0.0.1, 127.0.0.2, 127.0.0.3]

7. Create Cassandra User to restrict access to Cassandra cluster:

   CODE

   create role {username} with SUPERUSER = true AND LOGIN = true AND PASSWORD = '{password}';

For more information on common cassandra commands, please refer to Cassandra official documentation.