For few files in Perforce repository annotation doesn't show any data

[jkjha]

We have a opengrok v 1.12.6 running on docker. But for few files (just one file as of now that was reported), annotation data does not show anything. It just shows * in place of username (please check the screenshot):

When I see the web log, last message shows exit code 0:

<record>
  <date>2023-07-11T03:35:13.077694Z</date>
  <millis>1689046513077</millis>
  <nanos>694000</nanos>
  <sequence>428836</sequence>
  <logger>org.opengrok.indexer.util</logger>
  <level>FINE</level>
  <class>org.opengrok.indexer.util.Executor</class>
  <method>exec</method>
  <thread>18500</thread>
  <message>Finished command [/usr/bin/p4 print -q $'file_name.c@11212371'] in directory '/opengrok/src/<project_folder>' with exit code 0</message>

But before that there are few lines which mention about timeout and exit code 143:

<record>
  <date>2023-07-11T03:35:11.908470Z</date>
  <millis>1689046511908</millis>
  <nanos>470000</nanos>
  <sequence>428812</sequence>
  <logger>org.opengrok.indexer.util</logger>
  <level>WARNING</level>
  <class>org.opengrok.indexer.util.Executor$1</class>
  <method>run</method>
  <thread>24766</thread>
  <message>Terminating process of command [/usr/bin/p4 annotate -qci $'file_name.c@11212371'] in directory '/opengrok/src/project_folder' due to timeout 30 seconds</message>

----snip---
  <date>2023-07-11T03:35:11.910674Z</date>
  <millis>1689046511910</millis>
  <nanos>674000</nanos>
  <sequence>428814</sequence>
  <logger>org.opengrok.indexer.util</logger>
  <level>FINE</level>
  <class>org.opengrok.indexer.util.Executor</class>
  <method>exec</method>
  <thread>18500</thread>
  <message>Finished command [/usr/bin/p4 annotate -qci $'file_name.c@11212371'] in directory '/opengrok/src/<project_folder>'' with exit code 143</message>

This is happening only with this file, for other files in same project, it works fine. Can you please help here.

[jkjha]

I see a few suggestions here related to 143 exit code, but not sure what changes I should make in opengrok to solve this:

https://stackoverflow.com/questions/42972908/container-killed-by-the-applicationmaster-exit-code-is-143

I have following setting in setenv.sh file for tomcat:

JAVA_OPTS="$JAVA_OPTS -Xmx32g"
and below setting in sync.yml for memory:

- command:
    args: [opengrok-reindex-project,
      --api_timeout, '%API_TIMEOUT%',
       -J=-XX:-UseGCOverheadLimit, -J=-Xmx32g, -J=-server,
       --jar, /opengrok/lib/opengrok.jar,
       -t, /opengrok/etc/logging.properties.template, -p, '%PROJ%', -d, /opengrok/log/%PROJECT%,
       -U, '%URL%', -P, '%PROJECT%', --,
      --connectTimeout, '%API_TIMEOUT%',
      -r, dirbased, -G, -m, '4096', --leadingWildCards, 'on',
      -c, /usr/local/bin/ctags, -U, '%URL%', -o, /opengrok/etc/ctags.config, -H, '%PROJECT%']
    limits: {RLIMIT_NOFILE: 4096}
    args_subst: {"%API_TIMEOUT%": "$API_TIMEOUT"}

[vladak]

The cause is very clearly visible in the log message:

Terminating process of command [/usr/bin/p4 annotate ...'] in directory '/opengrok/src/project_folder' due to timeout 30 seconds

The webapp terminated the long running command because it took longer than the configured timeout for running such commands.

Before going further, it is important to determine whether the command will eventually finish within some longer time period or whether it is stuck (for example because it is trying to communicate over network that is black holed).

To do that, try to replicate the Tomcat environment and run the command by hand (ideally same environment variables, current working directory, etc.)

If this is the former, then raising the timeout in the configuration will help. To do that, it is necessary to use read-only configuration, specifically the interactiveCommandTimeout (see https://github.com/oracle/opengrok/wiki/Webapp-configuration).

Keep in mind that in order to take this effect, the webapp has to be restarted once the read-only configuration was merged into the configuration file (the default is /var/opengrok/etc/configuration.xml).

[jkjha]

I have added below configuration to read-only.xml... will be deploying this weekend on production... please let me know if the syntax is not correct:

  <void property="interactiveCommandTimeout">
   <int>300</int>
  </void>

[vladak]

Yep, this snippet looks good. Did you try to execute the command (in given directory, with given environment) by hand ?

[jkjha]

yes, it works manually ... I basically went to the container inside the directory and ran same command... it worked.

[jkjha]

may be when a bunch of command inside the container, its putting the load or causing timeout? But wondering why this happens with only this file... for other files (even bigger) it works well. I will try to debug this more.