Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Problem

When logging in to the Web Interface, this error is displayed:

or

the HubRing service is not started. This can be checked in the command line with:

Code Block
#> ps aux | grep hub | grep -v grep
#>

or

checking the HubRing's log file with

Code Block
#> tail -f /var/log/hubring/hubring.log

displays the following messages:

Code Block
...
[11649] 21 Jun 04:26:41 * Server started, HubRing version 1.0.0
[11649] 21 Jun 04:26:41 * Mixed Persistence: SUCCESS on loading dump_2014-06-20.13.44.24_8.rdb
[11649] 21 Jun 04:26:41 # Bad file format reading the append only file: make a backup of your AOF file, then use ./hr-check-aof --fix <filename>
...

Solution

The cause for this behavior is the lack of disk space or a possible disk problem.

All the cache files are kept into the /var/lib/hubring folder. At some point the volume that keeps the /var folder was out of free space. Thus, the cache system was unable to update cache files and corrupted them.

To fix this you'll need to clear some space and to make sure that you will not end up in this situation again.

After you have done that, you have to fix the damaged .aof file by following these steps:

STEP 1: After you receive the log message again, find out what .aof file is damaged using this mental exercise: look at the .rdb file from the message above (in our example dump_2014-06-20.13.44.24_8.rdb), increment the number before .rdb with 1 and and change the extension to .aof. The result is the corrupted file. For our example the damaged .aof will be:

Code Block
dump_2014-06-20.13.44.24_9.aof

STEP 2: Run the following command to fix the damaged file:

Code Block
#> hr-check-aof --fix <damaged aof filename>

Replace <damaged aof filename> with the name of the .aof file.

e.g. For our example it would be:

Code Block
#> hr-check-aof --fix <dump_2014-06-20.13.44.24_9.aof>

The result should be something like this:

Code Block
0x0002bffc: Expected to read 5 bytes, got 4 bytes
This will shrink the AOF from 180224 bytes, with 189 bytes, to 180035 bytes
Continue? [y/N]: y
Successfully truncated AOF

STEP 3: You are now able to start HubRing again, using this command:

Code Block
#>/var/lib/hubring # service hubring start

STEP 4: Check HubRing's status:

Code Block
#> service hubring status

You should see something like this:

Code Block
hubring (pid <pid number>) is running...

After that you'll be able to log in again and also to have calls.

Content by Label
showLabelsfalse
max5
showSpacefalse
sortmodified
reversetrue
typepage
cqllabel in ("validation","space","logging","disk","hubring") and space = currentSpace()
labelsvalidation logging hubring disk space

Except where otherwise noted, content in this space is licensed under a Creative Commons Attribution 4.0 International.