java.lang.UnsatisfiedLinkError: Error looking up function on Mac

[renxiawang]

Reproduce:

1. Compile the lib on mac:

g++ $CFLAGS $CPPFLAGS $CXXFLAGS -shared -fPIC \
  cldutil.cc cldutil_shared.cc compact_lang_det.cc compact_lang_det_hint_code.cc \
  compact_lang_det_impl.cc  debug.cc fixunicodevalue.cc \
  generated_entities.cc  generated_language.cc generated_ulscript.cc  \
  getonescriptspan.cc lang_script.cc offsetmap.cc  scoreonescriptspan.cc \
  tote.cc utf8statetable.cc  \
  cld_generated_cjk_uni_prop_80.cc cld2_generated_cjk_compatible.cc  \
  cld_generated_cjk_delta_bi_4.cc generated_distinct_bi_0.cc  \
  cld2_generated_quadchrome_2.cc cld2_generated_deltaoctachrome.cc \
  cld2_generated_distinctoctachrome.cc  cld_generated_score_quad_octa_2.cc  \
  -o libcld2.dylib $LDFLAGS -Wl,-install_name,libcld2.dylib

g++ $CFLAGS $CPPFLAGS $CXXFLAGS -shared -fPIC \
  cldutil.cc cldutil_shared.cc compact_lang_det.cc compact_lang_det_hint_code.cc \
  compact_lang_det_impl.cc  debug.cc fixunicodevalue.cc \
  generated_entities.cc  generated_language.cc generated_ulscript.cc  \
  getonescriptspan.cc lang_script.cc offsetmap.cc  scoreonescriptspan.cc \
  tote.cc utf8statetable.cc  \
  cld_generated_cjk_uni_prop_80.cc cld2_generated_cjk_compatible.cc  \
  cld_generated_cjk_delta_bi_32.cc generated_distinct_bi_0.cc  \
  cld2_generated_quad0122.cc cld2_generated_deltaocta0122.cc \
  cld2_generated_distinctocta0122.cc  cld_generated_score_quad_octa_0122.cc  \
  -o libcld2_full.dylib $LDFLAGS -Wl,-install_name,libcld2_full.dylib

2. Copy the *.dylib files to my java project resources folder:

• darwin/libcld2.dylib

3. See exception:

An exception or error caused a run to abort: Error looking up function '_ZN4CLD224ExtDetectLanguageSummaryEPKcibPKNS_8CLDHintsEiPNS_8LanguageEPiPdPSt6vectorINS_11ResultChunkESaISA_EES7_Pb': dlsym(0x7fd6d459d270, _ZN4CLD224ExtDetectLanguageSummaryEPKcibPKNS_8CLDHintsEiPNS_8LanguageEPiPdPSt6vectorINS_11ResultChunkESaISA_EES7_Pb): symbol not found 
java.lang.UnsatisfiedLinkError: Error looking up function '_ZN4CLD224ExtDetectLanguageSummaryEPKcibPKNS_8CLDHintsEiPNS_8LanguageEPiPdPSt6vectorINS_11ResultChunkESaISA_EES7_Pb': dlsym(0x7fd6d459d270, _ZN4CLD224ExtDetectLanguageSummaryEPKcibPKNS_8CLDHintsEiPNS_8LanguageEPiPdPSt6vectorINS_11ResultChunkESaISA_EES7_Pb): symbol not found
	at com.sun.jna.Function.<init>(Function.java:245)
	at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:566)
	at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:542)
	at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:528)
	at com.sun.jna.Library$Handler.invoke(Library.java:228)
	at org.commoncrawl.langdetect.cld2.$Proxy4._ZN4CLD224ExtDetectLanguageSummaryEPKcibPKNS_8CLDHintsEiPNS_8LanguageEPiPdPSt6vectorINS_11ResultChunkESaISA_EES7_Pb(Unknown Source)
	at org.commoncrawl.langdetect.cld2.Cld2.detect(Cld2.java:130)
	at org.commoncrawl.langdetect.cld2.Cld2.detect(Cld2.java:82)
	at org.commoncrawl.langdetect.cld2.Cld2.detect(Cld2.java:69)
        at my.class

when running code in Scala:

val result = Cld2.detect("Dapatkah anda membantu saya? Dapatkah anda membantu saya? Dapatkah anda membantu saya? Dapatkah anda membantu saya? Dapatkah anda membantu saya? Dapatkah anda membantu saya? Dapatkah anda membantu saya? Dapatkah anda membantu saya?")

[sebastian-nagel]

Sorry, I have no experience with linking Java against native libraries on Darwin and have no such system at hand to reproduce the issue.

• could be that name mangling works different. How are the functions called in the shared object? On Linux this could be checked by

% nm -D /usr/lib/x86_64-linux-gnu/libcld2.so.0.0.197
0000000000022c40 T _ZN4CLD224ExtDetectLanguageSummaryEPKcibPKNS_8CLDHintsEiPNS_8LanguageEPiPdPSt6vectorINS_11ResultChunkESaISA_EES7_Pb
0000000000022a00 T _ZN4CLD224ExtDetectLanguageSummaryEPKcibPNS_8LanguageEPiS4_Pb
0000000000022b20 T _ZN4CLD224ExtDetectLanguageSummaryEPKcibS1_iNS_8LanguageEPS2_PiPdS4_Pb
0000000000022a90 T _ZN4CLD224ExtDetectLanguageSummaryEPKcibS1_iNS_8LanguageEPS2_PiS4_Pb
0000000000022ba0 T _ZN4CLD233ExtDetectLanguageSummaryCheckUTF8EPKcibPKNS_8CLDHintsEiPNS_8LanguageEPiPdPSt6vectorINS_11ResultChunkESaISA_EES7_PbS7_

• is the library location (darwin/) set as java.library.path?

[renxiawang]

@sebastian-nagel you are right about the name mangling, I tried the code on Linux, where I eventually want my code to run on, and it works. The functions are called differently on mac.

[sebastian-nagel]

Thanks, @renxiawang, for digging into it to find the reason. Looks like there is no trivial solution and it would add an extra level of indirection:

• either wrap the CLD2 library into another native lib which exposes unmangled C-style function names. But then we could use also JNI to call to the native code. The advantage of JNA is only the packaged CDL2 library and the jar need to be installed resp. deployed.
• alternatively, could add platform-dependent function calls in Java
  Or is there a better solution?
  In doubt, I'd solve this by documentation - it's only 10 lines in Cld2Library and Cld2 which need to be changed.

[sebastian-nagel]

Closing because currently I'm not able to support platforms other than Linux. Documentation about potential issues such as name mangling is added. Thanks, @renxiawang!

[sebastian-nagel]

See also #7.