Cassandra 4.0.1 can't be started using 'cassandra -f' command on MAC OSX-CodePudding

After my Mac upgraded to Monterey, I had to reinstall cassandra from 3.x.x to 4.0.1.

I can't start Cassandra 4.0.1 using 'cassandra -f' command. I see following warning/errors:

WARN  [main] 2022-01-19 17:11:58,324 StartupChecks.java:143 - jemalloc shared library could not be preloaded to speed up memory allocations
WARN  [main] 2022-01-19 17:11:58,325 StartupChecks.java:187 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.

Errors:

ERROR [SSTableBatchOpen:12] 2022-01-19 17:12:02,616 DebuggableThreadPoolExecutor.java:263 - Error in ThreadPoolExecutor
java.lang.AssertionError: Stats component is missing for sstable /usr/local/var/lib/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/nb-346-big
    at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:461)
    at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:372)
    at org.apache.cassandra.io.sstable.format.SSTableReader$2.run(SSTableReader.java:540)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)

WARN  [main] 2022-01-19 17:12:07,595 CommitLogReplayer.java:305 - Origin of 6 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,598 CommitLogReplayer.java:305 - Origin of 4 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,599 CommitLogReplayer.java:305 - Origin of 4 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,600 CommitLogReplayer.java:305 - Origin of 6 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,601 CommitLogReplayer.java:305 - Origin of 3 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,602 CommitLogReplayer.java:305 - Origin of 2 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,602 CommitLogReplayer.java:305 - Origin of 1 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,603 CommitLogReplayer.java:305 - Origin of 3 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,604 CommitLogReplayer.java:305 - Origin of 1 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,604 CommitLogReplayer.java:305 - Origin of 1 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,604 CommitLogReplayer.java:305 - Origin of 3 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,606 CommitLogReplayer.java:305 - Origin of 677 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,607 CommitLogReplayer.java:305 - Origin of 2 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,607 CommitLogReplayer.java:305 - Origin of 6 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,608 CommitLogReplayer.java:305 - Origin of 1 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
WARN  [main] 2022-01-19 17:12:07,608 CommitLogReplayer.java:305 - Origin of 2 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored

WARN  [MemtableFlushWriter:1] 2022-01-19 17:12:08,879 NativeLibrary.java:318 - open(/usr/local/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f, O_RDONLY) failed, errno (24).
WARN  [MemtableFlushWriter:2] 2022-01-19 17:12:08,879 NativeLibrary.java:318 - open(/usr/local/var/lib/cassandra/data/system_schema/types-5a8b1ca866023f77a0459273d308917a, O_RDONLY) failed, errno (24).
ERROR [MemtableFlushWriter:1] 2022-01-19 17:12:08,885 LogReplica.java:108 - Failed to sync file /usr/local/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb_txn_flush_fd43a060-798d-11ec-901b-455a45092af6.log
org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException: /usr/local/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb_txn_flush_fd43a060-798d-11ec-901b-455a45092af6.log: Too many open files
    at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:867)
    at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java:810)
    at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java:104)
    at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogReplicaSet.java:225)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:138)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:128)
    at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSet.java:225)
    at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:363)
    at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:282)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:138)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:128)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:123)
    at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTransaction.java:466)
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.abort(Transactional.java:141)
    at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(LifecycleTransaction.java:250)
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.abort(Transactional.java:141)
    at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1140)
    at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1075)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.file.FileSystemException: /usr/local/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb_txn_flush_fd43a060-798d-11ec-901b-455a45092af6.log: Too many open files
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
    at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:844)
    ... 21 common frames omitted

ERROR [MemtableFlushWriter:1] 2022-01-19 17:12:08,887 DefaultFSErrorHandler.java:104 - Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException: /usr/local/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb_txn_flush_fd43a060-798d-11ec-901b-455a45092af6.log: Too many open files
    at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:867)
    at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java:810)
    at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java:104)
    at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogReplicaSet.java:225)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:138)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:128)
    at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSet.java:225)
    at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:363)
    at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:282)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:138)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:128)
    at org.apache.cassandra.utils.Throwables.perform(Throwables.java:123)
    at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTransaction.java:466)
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.abort(Transactional.java:141)
    at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(LifecycleTransaction.java:250)
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.abort(Transactional.java:141)
    at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1140)
    at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1075)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.file.FileSystemException: /usr/local/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb_txn_flush_fd43a060-798d-11ec-901b-455a45092af6.log: Too many open files
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
    at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:844)
    ... 21 common frames omitted

ERROR [MemtableFlushWriter:2] 2022-01-19 17:12:08,888 LogTransaction.java:304 - [nb_txn_flush_fd43a061-798d-11ec-901b-455a45092af6.log in /usr/local/var/lib/cassandra/data/system_schema/types-5a8b1ca866023f77a0459273d308917a] was not completed, trying to abort it now

It can be started using command 'brew services start cassandra' OR at least I see success message after starting it using 'brew services ...'

But while connecting it with using cqlsh, I see following messages:

bin % cqlsh
/usr/local/Cellar/cassandra/4.0.1/libexec/bin/cqlsh.py:460: DeprecationWarning: Legacy execution parameters will be removed in 4.0. Consider using execution profiles.
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': ConnectionRefusedError(61, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

I spent almost two days by using Java 8 then Java 11, tried cassandra@3 but nothing is working.

Any idea?

CodePudding user response：

The error is here: Too many open files <- you need to increase the limit on the number of open files. This could be done with ulimit command and make it permanent as described in this answer.