Problem

For some reasons it could be the case that you will see invalid checker timestamps in the cmc.log and the checkmk gui:

2022-04-28 11:45:33 [4] [core 25015] [smartping scheduler] refuse to schedule invalid jobhost "testhost-a" at 1970-01-01 01:00:00 with an invalid timestamp
2022-04-28 11:45:33 [4] [core 25015] [smartping scheduler] refuse to schedule invalid jobhost "testhost-b" at 1970-01-01 01:00:00 with an invalid timestamp
2022-04-28 11:45:33 [4] [core 25015] [smartping scheduler] refuse to schedule invalid jobhost "testhost-c" at 1970-01-01 01:00:00 with an invalid timestamp

In <2.1 we're there is no way to debug that due to the format of the state file. With Checkmk 2.1 the state file will have a different format. Anyway: If you face that issue in <2.1 you can open a case for that.

Solution

  1. Log on to the affected site

    su - mysite


  2. navigate to ~/var/check_mk/core

    OMD[mysite]:~$ cd ~/var/check_mk/core/


  3. here you will find all the "state" files

    OMD[mysite]:~/var/check_mk/core$ ls
archive/                 helper_config/   state.1  state.5  state.9
autochecks/              history          state.2  state.6
config                   state            state.3  state.7
discovered_host_labels/  state-version-7  state.4  state.8

    The cmc is using always the file "state"

  4. Now rename the "state" file to state.(number)

    BE  AWARE: all ad hoc Downtimes + Acknowledgements + Comments are gone, but you can restore the old state file later.

    The state is creating every 10 minutes or after changes in the GUI.

    A backup is created when the new file is 5% smaller than the old one. Checkmk will save 30 state files and then overwrite the old ones.


    OMD[mysite]:~/var/check_mk/core$ mv state state.10

    The number depends on how many state.(number) files you already have.


  5. You can either restart the whole site with

    OMD[mysite]:~ omd restart

    or do

    cmk -O


    Now a new "state" file is created!

Restore a state file


Go back to Step 4 of Solution and move  the state.NN to state


mv state.NN state


Don't restart the core in that case!





Related articles

Related articles appear here based on the labels you select. Click to edit the macro and add or change labels.



Related issues