[Using Sakai] Sakai Error on one of our two nodes
Anders Nordkvist
anders.nordqvist at his.se
Mon Sep 15 04:28:31 PDT 2014
Hi,
The two nodes are on two different virtual servers. This is a sample of what I get when I run your command (there is a lot of indexwork files but maybe this is normal):
java 10025 sakai 1892r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
java 10025 sakai 1893r REG 252,0 1100032145 279893 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.prx
java 10025 sakai 1894r REG 252,0 1100032145 279893 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.prx
java 10025 sakai 1895r REG 252,0 77395431 279880 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdt
java 10025 sakai 1896r REG 252,0 77395431 279880 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdt
java 10025 sakai 1897r REG 252,0 2306468 279882 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdx
java 10025 sakai 1898r REG 252,0 4612932 279857 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvx
java 10025 sakai 1899r REG 252,0 883806 279861 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvd
java 10025 sakai 1900r REG 252,0 486730548 279867 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvf
java 10025 sakai 1901r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
java 10025 sakai 1902r REG 252,0 2306468 279882 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdx
java 10025 sakai 1903r REG 252,0 4612932 279857 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvx
java 10025 sakai 1904r REG 252,0 883806 279861 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvd
java 10025 sakai 1905r REG 252,0 486730548 279867 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvf
java 10025 sakai 1906r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
java 10025 sakai 1907r REG 252,0 171385592 279889 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tis
java 10025 sakai 1908r REG 252,0 77395431 279880 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdt
java 10025 sakai 1909r REG 252,0 194976896 279892 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.frq
java 10025 sakai 1910r REG 252,0 1100032145 279893 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.prx
java 10025 sakai 1911r REG 252,0 171385592 279889 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tis
java 10025 sakai 1912r REG 252,0 194976896 279892 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.frq
java 10025 sakai 1913r REG 252,0 1100032145 279893 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.prx
java 10025 sakai 1914r REG 252,0 77395431 279880 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdt
java 10025 sakai 1915r REG 252,0 2306468 279882 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdx
java 10025 sakai 1916r REG 252,0 4612932 279857 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvx
java 10025 sakai 1917r REG 252,0 883806 279861 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvd
java 10025 sakai 1918r REG 252,0 486730548 279867 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvf
java 10025 sakai 1919r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
java 10025 sakai 1920r REG 252,0 77395431 279880 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdt
java 10025 sakai 1921r REG 252,0 2306468 279882 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdx
java 10025 sakai 1922r REG 252,0 4612932 279857 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvx
java 10025 sakai 1923r REG 252,0 883806 279861 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvd
java 10025 sakai 1924r REG 252,0 486730548 279867 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvf
java 10025 sakai 1925r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
java 10025 sakai 1926r REG 252,0 2306468 279882 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdx
java 10025 sakai 1927r REG 252,0 171385592 279889 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tis
java 10025 sakai 1928r REG 252,0 194976896 279892 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.frq
java 10025 sakai 1929r REG 252,0 1100032145 279893 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.prx
java 10025 sakai 1930r REG 252,0 77395431 279880 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdt
java 10025 sakai 1931r REG 252,0 2306468 279882 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdx
java 10025 sakai 1932r REG 252,0 4612932 279857 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvx
java 10025 sakai 1933r REG 252,0 883806 279861 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvd
java 10025 sakai 1934r REG 252,0 486730548 279867 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvf
java 10025 sakai 1935r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
java 10025 sakai 1936r REG 252,0 4612932 279857 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvx
java 10025 sakai 1937r REG 252,0 883806 279861 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvd
java 10025 sakai 1938r REG 252,0 486730548 279867 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvf
java 10025 sakai 1939r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
java 10025 sakai 1940u IPv4 683085 0t0 TCP scio2.hs.local:http-alt->193.10.178.7:62743 (ESTABLISHED)
java 10025 sakai 1941r REG 252,0 1100032145 279893 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.prx
java 10025 sakai 1942r REG 252,0 77395431 279880 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdt
java 10025 sakai 1943r REG 0,20 86080475 2269006 /data/sakai_datastore/vol2/2013/347/15/025720ff-29a1-455c-b32b-06423f29c16b (I have removed this IP:/volumes/storage/sp2010)
java 10025 sakai 1944r REG 252,0 2306468 279882 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdx
java 10025 sakai 1945r REG 252,0 4612932 279857 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvx
java 10025 sakai 1946r REG 252,0 883806 279861 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvd
java 10025 sakai 1947r REG 252,0 486730548 279867 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvf
java 10025 sakai 1948r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
java 10025 sakai 1949r REG 252,0 171385592 279889 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tis
java 10025 sakai 1950r REG 252,0 194976896 279892 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.frq
java 10025 sakai 1951r REG 252,0 1100032145 279893 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.prx
java 10025 sakai 1952r REG 252,0 77395431 279880 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdt
java 10025 sakai 1953r REG 252,0 2306468 279882 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdx
java 10025 sakai 1954r REG 252,0 4612932 279857 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvx
java 10025 sakai 1955r REG 252,0 883806 279861 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvd
java 10025 sakai 1956r REG 252,0 486730548 279867 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvf
java 10025 sakai 1957r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
java 10025 sakai 1958u IPv4 680765 0t0 TCP scio2.hs.local:http-alt->IP:port (ESTABLISHED)
java 10025 sakai 1959u IPv4 684330 0t0 TCP scio2.hs.local:PORT->hsdc1.hs.local:ldaps (ESTABLISHED)
java 10025 sakai 1960r REG 252,0 171385592 279889 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tis
java 10025 sakai 1961r REG 252,0 194976896 279892 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.frq
java 10025 sakai 1962r REG 252,0 1100032145 279893 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.prx
java 10025 sakai 1963r REG 252,0 77395431 279880 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdt
java 10025 sakai 1964r REG 252,0 2306468 279882 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdx
java 10025 sakai 1965r REG 252,0 4612932 279857 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvx
java 10025 sakai 1966r REG 252,0 883806 279861 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvd
java 10025 sakai 1967r REG 252,0 486730548 279867 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvf
java 10025 sakai 1968r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
java 10025 sakai 1969u IPv4 680868 0t0 TCP scio2.hs.local:PORT->hsdc1.hs.local:ldaps (ESTABLISHED)
java 10025 sakai 1970r REG 0,20 86080475 2269006 /data/sakai_datastore/vol2/2013/347/15/025720ff-29a1-455c-b32b-06423f29c16b (IP:/volumes/storage/sp2010)
java 10025 sakai 1972r REG 252,0 171385592 279889 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tis
java 10025 sakai 1974r REG 252,0 194976896 279892 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.frq
java 10025 sakai 1975r REG 252,0 1100032145 279893 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.prx
java 10025 sakai 1976r REG 252,0 77395431 279880 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdt
java 10025 sakai 1978r REG 252,0 2306468 279882 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.fdx
java 10025 sakai 1979r REG 252,0 4612932 279857 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvx
java 10025 sakai 1980r REG 252,0 883806 279861 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvd
java 10025 sakai 1981r REG 252,0 486730548 279867 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.tvf
java 10025 sakai 1982r REG 252,0 10090784 279832 /opt/tomcat-7.0.42/sakai/indexwork/index/_204.nrm
lsof 21744 sakai cwd DIR 0,20 19 2723969 /data/home/sakai (IP:/volumes/storage/sp2010)
lsof 21744 sakai rtd DIR 252,0 4096 2 /
lsof 21744 sakai txt REG 252,0 131312 1453896 /usr/bin/lsof
lsof 21744 sakai mem REG 252,0 52120 1573114 /lib/x86_64-linux-gnu/libnss_files-2.15.so
lsof 21744 sakai mem REG 252,0 47680 1573118 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
lsof 21744 sakai mem REG 252,0 97248 1574489 /lib/x86_64-linux-gnu/libnsl-2.15.so
lsof 21744 sakai mem REG 252,0 35680 1573112 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
lsof 21744 sakai mem REG 252,0 2919792 1447543 /usr/lib/locale/locale-archive
lsof 21744 sakai mem REG 252,0 1811128 1573110 /lib/x86_64-linux-gnu/libc-2.15.so
lsof 21744 sakai mem REG 252,0 149280 1574480 /lib/x86_64-linux-gnu/ld-2.15.so
lsof 21744 sakai mem REG 252,0 26258 1465283 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
lsof 21744 sakai 0u CHR 136,1 0t0 4 /dev/pts/1
lsof 21744 sakai 1w FIFO 0,8 0t0 683090 pipe
lsof 21744 sakai 2u CHR 136,1 0t0 4 /dev/pts/1
lsof 21744 sakai 3r DIR 0,3 0 1 /proc
lsof 21744 sakai 4r DIR 0,3 0 682516 /proc/21744/fd
lsof 21744 sakai 5w FIFO 0,8 0t0 682521 pipe
lsof 21744 sakai 6r FIFO 0,8 0t0 682522 pipe
grep 21745 sakai cwd DIR 0,20 19 2723969 /data/home/sakai (IP:/volumes/storage/sp2010)
grep 21745 sakai rtd DIR 252,0 4096 2 /
grep 21745 sakai txt REG 252,0 159288 1966093 /bin/grep
grep 21745 sakai mem REG 252,0 1976 132382 /usr/share/locale-langpack/en_GB/LC_MESSAGES/grep.mo
grep 21745 sakai mem REG 252,0 26258 1465283 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
grep 21745 sakai mem REG 252,0 2919792 1447543 /usr/lib/locale/locale-archive
grep 21745 sakai mem REG 252,0 1811128 1573110 /lib/x86_64-linux-gnu/libc-2.15.so
grep 21745 sakai mem REG 252,0 14768 1574484 /lib/x86_64-linux-gnu/libdl-2.15.so
grep 21745 sakai mem REG 252,0 149280 1574480 /lib/x86_64-linux-gnu/ld-2.15.so
grep 21745 sakai 0r FIFO 0,8 0t0 683090 pipe
grep 21745 sakai 1u CHR 136,1 0t0 4 /dev/pts/1
grep 21745 sakai 2u CHR 136,1 0t0 4 /dev/pts/1
lsof 21746 sakai cwd DIR 0,20 19 2723969 /data/home/sakai (IP:/volumes/storage/sp2010)
lsof 21746 sakai rtd DIR 252,0 4096 2 /
lsof 21746 sakai txt REG 252,0 131312 1453896 /usr/bin/lsof
lsof 21746 sakai mem REG 252,0 52120 1573114 /lib/x86_64-linux-gnu/libnss_files-2.15.so
lsof 21746 sakai mem REG 252,0 47680 1573118 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
lsof 21746 sakai mem REG 252,0 97248 1574489 /lib/x86_64-linux-gnu/libnsl-2.15.so
lsof 21746 sakai mem REG 252,0 35680 1573112 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
lsof 21746 sakai mem REG 252,0 2919792 1447543 /usr/lib/locale/locale-archive
lsof 21746 sakai mem REG 252,0 1811128 1573110 /lib/x86_64-linux-gnu/libc-2.15.so
lsof 21746 sakai mem REG 252,0 149280 1574480 /lib/x86_64-linux-gnu/ld-2.15.so
lsof 21746 sakai 4r FIFO 0,8 0t0 682521 pipe
lsof 21746 sakai 7w FIFO 0,8 0t0 682522 pipe
su 28574 sakai cwd unknown /proc/28574/cwd (readlink: Permission denied)
su 28574 sakai rtd unknown /proc/28574/root (readlink: Permission denied)
su 28574 sakai txt unknown /proc/28574/exe (readlink: Permission denied)
su 28574 sakai NOFD /proc/28574/fd (opendir: Permission denied)
bash 28581 sakai cwd DIR 252,0 53248 263838 /opt/tomcat-7.0.42/logs
bash 28581 sakai rtd DIR 252,0 4096 2 /
bash 28581 sakai txt REG 252,0 959120 1972162 /bin/bash
bash 28581 sakai mem REG 252,0 52120 1573114 /lib/x86_64-linux-gnu/libnss_files-2.15.so
bash 28581 sakai mem REG 252,0 47680 1573118 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
bash 28581 sakai mem REG 252,0 97248 1574489 /lib/x86_64-linux-gnu/libnsl-2.15.so
bash 28581 sakai mem REG 252,0 35680 1573112 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
bash 28581 sakai mem REG 252,0 2919792 1447543 /usr/lib/locale/locale-archive
bash 28581 sakai mem REG 252,0 1811128 1573110 /lib/x86_64-linux-gnu/libc-2.15.so
bash 28581 sakai mem REG 252,0 14768 1574484 /lib/x86_64-linux-gnu/libdl-2.15.so
bash 28581 sakai mem REG 252,0 159200 1572932 /lib/x86_64-linux-gnu/libtinfo.so.5.9
bash 28581 sakai mem REG 252,0 149280 1574480 /lib/x86_64-linux-gnu/ld-2.15.so
bash 28581 sakai mem REG 252,0 32228 132270 /usr/share/locale-langpack/en_GB/LC_MESSAGES/bash.mo
bash 28581 sakai mem REG 252,0 26258 1465283 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
bash 28581 sakai 0u CHR 136,2 0t0 5 /dev/pts/2
bash 28581 sakai 1u CHR 136,2 0t0 5 /dev/pts/2
bash 28581 sakai 2u CHR 136,2 0t0 5 /dev/pts/2
bash 28581 sakai 255u CHR 136,2 0t0 5 /dev/pts/2
less 29515 sakai cwd DIR 252,0 53248 263838 /opt/tomcat-7.0.42/logs
less 29515 sakai rtd DIR 252,0 4096 2 /
less 29515 sakai txt REG 252,0 149384 1966157 /bin/less
less 29515 sakai mem REG 252,0 2919792 1447543 /usr/lib/locale/locale-archive
less 29515 sakai mem REG 252,0 159200 1572932 /lib/x86_64-linux-gnu/libtinfo.so.5.9
less 29515 sakai mem REG 252,0 14768 1574484 /lib/x86_64-linux-gnu/libdl-2.15.so
less 29515 sakai mem REG 252,0 1811128 1573110 /lib/x86_64-linux-gnu/libc-2.15.so
less 29515 sakai mem REG 252,0 133808 1572928 /lib/x86_64-linux-gnu/libncurses.so.5.9
less 29515 sakai mem REG 252,0 149280 1574480 /lib/x86_64-linux-gnu/ld-2.15.so
less 29515 sakai mem REG 252,0 26258 1465283 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
less 29515 sakai 0u CHR 136,2 0t0 5 /dev/pts/2
less 29515 sakai 1u CHR 136,2 0t0 5 /dev/pts/2
less 29515 sakai 2u CHR 136,2 0t0 5 /dev/pts/2
less 29515 sakai 3r CHR 5,0 0t0 1037 /dev/tty
less 29515 sakai 4r REG 252,0 941561394 275341 /opt/tomcat-7.0.42/logs/catalina.out
su 31157 sakai cwd unknown /proc/31157/cwd (readlink: Permission denied)
su 31157 sakai rtd unknown /proc/31157/root (readlink: Permission denied)
su 31157 sakai txt unknown /proc/31157/exe (readlink: Permission denied)
su 31157 sakai NOFD /proc/31157/fd (opendir: Permission denied)
bash 31164 sakai cwd DIR 252,0 4096 264045 /opt/tomcat-7.0.42/sakai/indexwork
bash 31164 sakai rtd DIR 252,0 4096 2 /
bash 31164 sakai txt REG 252,0 959120 1972162 /bin/bash
bash 31164 sakai mem REG 252,0 52120 1573114 /lib/x86_64-linux-gnu/libnss_files-2.15.so
bash 31164 sakai mem REG 252,0 47680 1573118 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
bash 31164 sakai mem REG 252,0 97248 1574489 /lib/x86_64-linux-gnu/libnsl-2.15.so
bash 31164 sakai mem REG 252,0 35680 1573112 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
bash 31164 sakai mem REG 252,0 2919792 1447543 /usr/lib/locale/locale-archive
bash 31164 sakai mem REG 252,0 1811128 1573110 /lib/x86_64-linux-gnu/libc-2.15.so
bash 31164 sakai mem REG 252,0 14768 1574484 /lib/x86_64-linux-gnu/libdl-2.15.so
bash 31164 sakai mem REG 252,0 159200 1572932 /lib/x86_64-linux-gnu/libtinfo.so.5.9
bash 31164 sakai mem REG 252,0 149280 1574480 /lib/x86_64-linux-gnu/ld-2.15.so
bash 31164 sakai mem REG 252,0 32228 132270 /usr/share/locale-langpack/en_GB/LC_MESSAGES/bash.mo
bash 31164 sakai mem REG 252,0 26258 1465283 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
bash 31164 sakai 0u CHR 136,3 0t0 6 /dev/pts/3
bash 31164 sakai 1u CHR 136,3 0t0 6 /dev/pts/3
bash 31164 sakai 2u CHR 136,3 0t0 6 /dev/pts/3
bash 31164 sakai 255u CHR 136,3 0t0 6 /dev/pts/3
There only seems to be one java process on the node:
sakai at scio2:~$ ps -ef | grep -i 'java'
sakai 10025 1 14 08:19 pts/1 00:43:59 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.util.logging.config.file=/opt/tomcat-7.0.42/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -Xmx6144m -XX:MaxPermSize=1024m -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dsun.lang.ClassLoader.allowArraySyntax=true -Dfile.encoding=UTF-8 -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/opt/tomcat-7.0.42/endorsed -classpath /opt/tomcat-7.0.42/bin/bootstrap.jar:/opt/tomcat-7.0.42/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat-7.0.42 -Dcatalina.home=/opt/tomcat-7.0.42 -Djava.io.tmpdir=/opt/tomcat-7.0.42/temp org.apache.catalina.startup.Bootstrap start
sakai 20971 9847 0 13:19 pts/1 00:00:00 grep --color=auto -i java
From: Stephen Marquard [mailto:stephen.marquard at uct.ac.za]
Sent: den 15 september 2014 13:03
To: Anders Nordkvist; sakai-user at collab.sakaiproject.org
Subject: RE: Sakai Error on one of our two nodes
If you have more than one java process running, then that would be a factor. Are your 2 nodes on one server, or one node on two servers?
I'd suggest you take a look at:
lsof -u tomcat | grep -v jar
and see if there's anything unusual, and also add
ulimit -n 5000
to your Sakai startup script to see if that helps.
Cheers
Stephen
---
Stephen Marquard, Learning Technologies Co-ordinator,
Centre for Innovation in Learning and Teaching (CILT)
University of Cape Town
http://www.cilt.uct.ac.za
stephen.marquard at uct.ac.za<mailto:stephen.marquard at uct.ac.za>
Phone: +27-21-650-5037 Cell: +27-83-500-5290
From: Anders Nordkvist [mailto:anders.nordqvist at his.se]
Sent: 15 September 2014 12:58 PM
To: Stephen Marquard; sakai-user at collab.sakaiproject.org<mailto:sakai-user at collab.sakaiproject.org>
Subject: RE: Sakai Error on one of our two nodes
Hi Stephen,
Thanks for the tips. I get this when I run the commands:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63739
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 63739
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
sakai at scio2:~$ lsof -u sakai | wc -l
2769
If I understand this right we have a max of 1024 for open files a process but the actually open files are 2769. Is this because there is more processes running?
Regards Anders
From: Stephen Marquard [mailto:stephen.marquard at uct.ac.za]
Sent: den 15 september 2014 12:26
To: Anders Nordkvist; sakai-user at collab.sakaiproject.org<mailto:sakai-user at collab.sakaiproject.org>
Subject: RE: Sakai Error on one of our two nodes
Hi Anders
You have 2 different problems; one from "Too many open files" and the other from the search service.
For the "too many open files" issue, you should see how many are being used and what the OS limit is on your app server. For example if your Sakai process runs as the tomcat user, you can run:
# lsof -u tomcat | wc -l
3821
and run "ulimit -a" to see the per-process OS limits. You can change these in your Sakai startup script, e.g. we have:
# Increase max open files
ulimit -n 100000
which is probably totally unnecessarily large, but we definitely had to increase it past the default 1024 in the early days. 5000 is perhaps reasonable.
It's possible the "too many open files" is a symptom of another problem rather than just an underlying limit that you've run into, in which case you need to see what those open files are (which could include socket connections) and why they are getting opened and not closed.
Regards
Stephen
---
Stephen Marquard, Learning Technologies Co-ordinator,
Centre for Innovation in Learning and Teaching (CILT)
University of Cape Town
http://www.cilt.uct.ac.za
stephen.marquard at uct.ac.za<mailto:stephen.marquard at uct.ac.za>
Phone: +27-21-650-5037 Cell: +27-83-500-5290
From: sakai-user-bounces at collab.sakaiproject.org<mailto:sakai-user-bounces at collab.sakaiproject.org> [mailto:sakai-user-bounces at collab.sakaiproject.org] On Behalf Of Anders Nordkvist
Sent: 15 September 2014 12:07 PM
To: sakai-user at collab.sakaiproject.org<mailto:sakai-user at collab.sakaiproject.org>
Subject: [Using Sakai] Sakai Error on one of our two nodes
Hi,
We have had problems with Sakai at the University of Skövde Sweden after an OS update and restart of systems last friday. We have 2.9.x and have two Sakai nodes and on top of that we have a netscaler distributing the load and behind a mysql server. The Sakai nodes collect information via LDAP from our Microsoft AD. The problem occurred several hours after the update of OS and restart of machines (about 11hours). During this time you only have a 50/50 % chance to login because the netscaler is not working properly and is not directing traffic to the working node. Can you guys please take a look at this and see if you can figure it out? This is the log from the beginning:
2014-09-12 22:08:07,941 WARN http-bio-8080-exec-121 org.apache.myfaces.shared_impl.renderkit.html.HtmlImageRendererBase - ALT attribute is missing for : _idJsp64
2014-09-12 22:14:00,421 WARN http-bio-8080-exec-108 com.sun.faces.renderkit.html_basic.HtmlBasicRenderer - Unable to find component with ID 'df_compose_title' in view.
2014-09-12 22:14:00,422 WARN http-bio-8080-exec-108 com.sun.faces.renderkit.html_basic.HtmlBasicRenderer - Unable to find component with ID 'df_compose_body' in view.
Sep 12, 2014 10:17:00 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)
Sep 12, 2014 10:17:00 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)
Sep 12, 2014 10:17:00 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)
Sep 12, 2014 10:17:00 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)
Sep 12, 2014 10:17:31 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)
Later on I can see this:
2014-09-12 23:56:45,667 ERROR http-bio-8080-exec-121 edu.amc.sakai.user.JLDAPDirectoryProvider - getUser() failed [eid: b14verca]
LDAPException: Unable to connect to server hsdc1.hs.local:636 (91) Connect Error
java.net.SocketException: Too many open files
at com.novell.ldap.Connection.connect(Unknown Source)
at com.novell.ldap.Connection.connect(Unknown Source)
at com.novell.ldap.LDAPConnection.connect(Unknown Source)
at edu.amc.sakai.user.SimpleLdapConnectionManager.connect(SimpleLdapConnectionManager.java:244)
at edu.amc.sakai.user.SimpleLdapConnectionManager.getConnection(SimpleLdapConnectionManager.java:65)
at edu.amc.sakai.user.JLDAPDirectoryProvider.searchDirectory(JLDAPDirectoryProvider.java:954)
at edu.amc.sakai.user.JLDAPDirectoryProvider.searchDirectoryForSingleEntry(JLDAPDirectoryProvider.java:902)
at edu.amc.sakai.user.JLDAPDirectoryProvider.getUserByEid(JLDAPDirectoryProvider.java:824)
at edu.amc.sakai.user.JLDAPDirectoryProvider.getUserByEid(JLDAPDirectoryProvider.java:778)
at edu.amc.sakai.user.JLDAPDirectoryProvider.getUser(JLDAPDirectoryProvider.java:603)
at org.sakaiproject.user.impl.BaseUserDirectoryService.getProvidedUserByEid(BaseUserDirectoryService.java:656)
at org.sakaiproject.user.impl.BaseUserDirectoryService.getUser(BaseUserDirectoryService.java:722)
at org.sakaiproject.user.impl.BaseUserDirectoryService.getCurrentUser(BaseUserDirectoryService.java:890)
at org.sakaiproject.authz.impl.SakaiSecurity.unlock(SakaiSecurity.java:222)
at org.sakaiproject.authz.cover.SecurityService.unlock(SecurityService.java:91)
at org.sakaiproject.portal.charon.site.PortalSiteHelperImpl.pageListToMap(PortalSiteHelperImpl.java:583)
at org.sakaiproject.portal.charon.handlers.WorksiteHandler.includeWorksite(WorksiteHandler.java:195)
at org.sakaiproject.portal.charon.handlers.WorksiteHandler.doWorksite(WorksiteHandler.java:165)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doError(SkinnableCharonPortal.java:270)
at org.sakaiproject.portal.charon.handlers.PresenceHandler.doPresence(PresenceHandler.java:117)
at org.sakaiproject.portal.charon.handlers.PresenceHandler.doGet(PresenceHandler.java:70)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:894)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:680)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Too many open files
I can also see this and I don't know if its related:
00:00:13,955 ERROR IndexManager org.sakaiproject.search.indexer.impl.TransactionalIndexWorker - Failed to Add Documents
org.sakaiproject.search.transaction.api.IndexTransactionException: Cant Create Transaction Index working space
at org.sakaiproject.search.indexer.impl.IndexUpdateTransactionImpl.getInternalIndexWriter(IndexUpdateTransactionImpl.java:205)
at org.sakaiproject.search.indexer.impl.IndexUpdateTransactionImpl.getIndexWriter(IndexUpdateTransactionImpl.java:168)
at org.sakaiproject.search.indexer.impl.IndexUpdateTransactionImpl.getIndexReader(IndexUpdateTransactionImpl.java:338)
at org.sakaiproject.search.indexer.impl.TransactionalIndexWorker.processTransaction(TransactionalIndexWorker.java:229)
at org.sakaiproject.search.indexer.impl.TransactionalIndexWorker.process(TransactionalIndexWorker.java:132)
at org.sakaiproject.search.indexer.impl.ConcurrentSearchIndexBuilderWorkerImpl.runOnce(ConcurrentSearchIndexBuilderWorkerImpl.java:273)
at org.sakaiproject.search.journal.impl.IndexManagementTimerTask.run(IndexManagementTimerTask.java:137)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/opt/tomcat-7.0.42/sakai/indexwork/indexer-work/indextx-1410515455900/write.lock<mailto:NativeFSLock@/opt/tomcat-7.0.42/sakai/indexwork/indexer-work/indextx-1410515455900/write.lock>: java.io.FileNotFoundException: /opt/tomcat-7.0.42/sakai/indexwork/indexer-work/indextx-1410515455900/write.lock (Too many open files)
at org.apache.lucene.store.Lock.obtain(Lock.java:85)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1562)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1090)
at org.sakaiproject.search.indexer.impl.IndexUpdateTransactionImpl.getInternalIndexWriter(IndexUpdateTransactionImpl.java:194)
... 8 more
Caused by: java.io.FileNotFoundException: /opt/tomcat-7.0.42/sakai/indexwork/indexer-work/indextx-1410515455900/write.lock (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:183)
at org.apache.lucene.store.Lock.obtain(Lock.java:99)
... 11 more
Sep 14, 2014 12:00:15 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)00:00:13,955 ERROR IndexManager org.sakaiproject.search.indexer.impl.TransactionalIndexWorker - Failed to Add Documents
org.sakaiproject.search.transaction.api.IndexTransactionException: Cant Create Transaction Index working space
at org.sakaiproject.search.indexer.impl.IndexUpdateTransactionImpl.getInternalIndexWriter(IndexUpdateTransactionImpl.java:205)
at org.sakaiproject.search.indexer.impl.IndexUpdateTransactionImpl.getIndexWriter(IndexUpdateTransactionImpl.java:168)
at org.sakaiproject.search.indexer.impl.IndexUpdateTransactionImpl.getIndexReader(IndexUpdateTransactionImpl.java:338)
at org.sakaiproject.search.indexer.impl.TransactionalIndexWorker.processTransaction(TransactionalIndexWorker.java:229)
at org.sakaiproject.search.indexer.impl.TransactionalIndexWorker.process(TransactionalIndexWorker.java:132)
at org.sakaiproject.search.indexer.impl.ConcurrentSearchIndexBuilderWorkerImpl.runOnce(ConcurrentSearchIndexBuilderWorkerImpl.java:273)
at org.sakaiproject.search.journal.impl.IndexManagementTimerTask.run(IndexManagementTimerTask.java:137)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/opt/tomcat-7.0.42/sakai/indexwork/indexer-work/indextx-1410515455900/write.lock<mailto:NativeFSLock@/opt/tomcat-7.0.42/sakai/indexwork/indexer-work/indextx-1410515455900/write.lock>: java.io.FileNotFoundException: /opt/tomcat-7.0.42/sakai/indexwork/indexer-work/indextx-1410515455900/write.lock (Too many open files)
at org.apache.lucene.store.Lock.obtain(Lock.java:85)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1562)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1090)
at org.sakaiproject.search.indexer.impl.IndexUpdateTransactionImpl.getInternalIndexWriter(IndexUpdateTransactionImpl.java:194)
... 8 more
Caused by: java.io.FileNotFoundException: /opt/tomcat-7.0.42/sakai/indexwork/indexer-work/indextx-1410515455900/write.lock (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:183)
at org.apache.lucene.store.Lock.obtain(Lock.java:99)
... 11 more
Sep 14, 2014 12:00:15 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)
We restarted the server on Saturday about 10:05 and then about 20:33 we get it again:
2014-09-14 20:32:14,408 WARN http-bio-8080-exec-204 org.apache.myfaces.shared_impl.renderkit.html.HtmlImageRendererBase - ALT attribute is missing for : _idJsp64
2014-09-14 20:33:06,840 WARN http-bio-8080-exec-186 org.apache.myfaces.shared_impl.renderkit.html.HtmlImageRendererBase - ALT attribute is missing for : _idJsp64
Sep 14, 2014 8:33:32 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)
Sep 14, 2014 8:33:32 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)
Sep 14, 2014 8:33:32 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)
Sep 14, 2014 8:33:32 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:745)
Sep 14, 2014 8:33:32 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
I restarted again and now its working so far but Im afraid it will go down again in the evening when nobody is working.
PS. On the firs node that have worked all the time we got this error:
2014-09-14 00:02:33,230 ERROR IndexManager org.sakaiproject.search.optimize.shared.impl.DbJournalOptimizationManager - This node already merging shared segments, index writer scio1:1395755228716 This node is currently optimizing the shared segments, This is an error as only one copy of this node should be Active in the clustersee http://jira.sakaiproject.org/browse/SRCH-38
2014-09-14 00:02:43,230 ERROR IndexManager org.sakaiproject.search.optimize.shared.impl.DbJournalOptimizationManager - This node already merging shared segments, index writer scio1:1395755228716 This node is currently optimizing the shared segments, This is an error as only one copy of this node should be Active in the clustersee http://jira.sakaiproject.org/browse/SRCH-38
2014-09-14 00:02:53,230 ERROR IndexManager org.sakaiproject.search.optimize.shared.impl.DbJournalOptimizationManager - This node already merging shared segments, index writer scio1:1395755228716 This node is currently optimizing the shared segments, This is an error as only one copy of this node should be Active in the clustersee http://jira.sakaiproject.org/browse/SRCH-38
2014-09-14 00:03:03,230 ERROR IndexManager org.sakaiproject.search.optimize.shared.impl.DbJournalOptimizationManager - This node already merging shared segments, index writer scio1:1395755228716 This node is currently optimizing the shared segments, This is an error as only one copy of this node should be Active in the clustersee http://jira.sakaiproject.org/browse/SRCH-38
I updated the database on this one according the Jira with (committed on all rows on database_journal). Now The error is gone.
Regards
Anders Nordkvist
System administrator
University Of Skövde
Sweden
________________________________
UNIVERSITY OF CAPE TOWN
This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-user/attachments/20140915/8fa201c2/attachment-0001.html
More information about the sakai-user
mailing list