[MineSweeping] The Long Struggle of DensePose Installation

Update in October 2019:

DensePose has been re-implemented with the brand-new object detection framework Detectron2, which is based on PyTorch and much easier to install and use (You don't have to manually compile Caffe2)
I strongly recommend that you check out the new official DensePose code at https://github.com/facebookresearch/detectron2/tree/master/projects/DensePose.

DensePose is a great work in real-time human pose estimation, which is based on Caffe2 and Detectron framework. It extracts dense human body 3D surface based on RGB images. The installation instructions are provided here.

During my installation process, these are the problems that took me some time to tackle. I spent on week to finally figure out solutions to all the issues. So lucky of me not to give up too early...

By the way, before you suffer too much, I strongly recommend following the step-by-step Caffe2+DensePose installation guide by @Johnqczhang. If you think you're almost there, help yourself with the solutions below~

1 Environment

System: Ubuntu 18.04
Linux kernel: 4.15.0-29-generic
Graphics card: NVIDIA GeForce 1080Ti
Graphics driver: 410.48
CUDA: 10.0.130
cuDNN: 7.3.1
Caffe2: Built from source
Python: 2.7.15, based on Anaconda 4.5.11

2 Problems & Solutions

2.1 Caffe2 module not found

Details

Occurred when running make.

Main error message:

Could not find a package configuration file provided by "Caffe2" with any  
of the following names: 
    Caffe2Config.cmake 
    caffe2-config.cmake

Cause

Caffe2 build path isn't known by CMake.

Solution

Added one line in the beginning of CMakeLists.txt:

1	set(Caffe2_DIR "/path/to/pytorch/torch/share/cmake/Caffe2/")

(Note: set(Caffe2_DIR "/path/to/pytorch/build/") can also fix this issue but may cause other issues.)

2.2 Detectron ops lib not found

Details

Occurred when running python2 $DENSEPOSE/detectron/tests/test_spatial_narrow_as_op.py after make.

Main error message:

1	Detectron ops lib not found; make sure that your Caffe2 version includes Detectron module.

Cause

Seems that the Python part of DensePose couldn't recognize Caffe2.

Solution

Add /path/to/pytorch/build to PYTHONPATH environment variable. Could be added by directly export PYTHONPATH=$PYTHONPATH:/path/to/pytorch/build instruction or by adding this line to ~/.bashrc. Remember to run source ~/.bashrc after the modification.

2.3 *.cmake files not found & Unknown CMake command "caffe2_interface_library"

Details

Occurred when running make ops.

Main error message:

CMake Error at /path/to/pytorch/build/Caffe2Config.cmake:14 (include):
  include could not find load file:

    /path/to/pytorch/build/public/utils.cmake
    /path/to/pytorch/build/public/threads.cmake
    /path/to/pytorch/build/public/cuda.cmake
    /path/to/pytorch/build/public/mkl.cmake
    /path/to/pytorch/build/Caffe2Targets.cmake

Call Stack (most recent call first):
  CMakeLists.txt:8 (find_package)

CMake Error at /path/to/pytorch/build/Caffe2Config.cmake:117 (caffe2_interface_library):
  Unknown CMake command "caffe2_interface_library".
Call Stack (most recent call first):
  CMakeLists.txt:8 (find_package)

(Several *.cmake files, I only showed a few.)

Cause

These files are not in the pytorch/build directory. By searching, I found that they are in the pytorch/torch/share/cmake/Caffe2 directory.

Solution

Added one line in the beginning of CMakeLists.txt:

1	set(Caffe2_DIR "/path/to/pytorch/torch/share/cmake/Caffe2/")

2.4 "context_gpu.h" not found.

Details

Occurred when running make ops.

I forgot to record the error messages, but it should be obvious that some header files(not just context_gpu.h) are missing.

Cause

This time it's the include path not recognized...

Solution

Added one line in the beginning of CMakeLists.txt:

1	include_directories("/path/to/pytorch/torch/lib/include")

2.5 This file was generated by a newer version of protoc which is incompatible with your Protocol Buffer headers. Please update your headers.

Details

Occurred when running make ops.

Main error message:

/path/to/pytorch/torch/lib/include/caffe2/proto/caffe2.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is
#error This file was generated by a newer version of protoc which is
^
/path/to/pytorch/torch/lib/include/caffe2/proto/caffe2.pb.h:13:2: error: #error incompatible with your Protocol Buffer headers. Please update
#error incompatible with your Protocol Buffer headers. Please update
^
/path/to/pytorch/torch/lib/include/caffe2/proto/caffe2.pb.h:14:2: error: #error your headers.
#error your headers.
^

Cause

If you only have a protobuf higher than v3.6.1, this should not happen. Check if you have multiple protobufs installed from different sources. (In my case, there was a protobuf v3.2.0 installed with apt-get earlier)

Solution

I can't provide an exact solution. Please try

1	which protoc

and see where protobuf is installed. If this shows the protobuf you installed with Anaconda, remove it completely and try this again. Since DensePose tells you that you have an older version of protobuf, you should be able to locate one. After finding it, remove it or upgrade it to v3.6.1 or higher. I would prefer installing protobuf from source here. It's not so painful as installing DensePose.

2.6 "mkl_cblas.h" not found.

Details

Occurred when running make ops.

I forgot to record the error messages, but it should be obvious too.

Cause

Intel Math Kernel Library was turned on but not found. (Why is it enabled when I didn't even install it???)

Solution

Install Intel Math Kernel Library here and add /opt/intel/compilers_and_libraries_2019.1.144/linux/mkl/include to C_PATH environment variable:

1	export CPATH=$CPATH:/opt/intel/compilers_and_libraries_2019.1.144/linux/mkl/include

The exact path may vary according to the MKL version and your configuration.
Maybe try find / -name mkl_cblas.h to make sure of its location after the installation.

Adding the path to CMakeLists.txt should also be helpful, but I didn't test it:

1	include_directories("/opt/intel/compilers_and_libraries_2019.1.144/Linux/mkl/include")

2.7 GetSingleArgument’ is not a member of ‘caffe2::PoolPointsInterpOp’

Details

Occurred when running make ops.
Main error message:

/path/to/pytorch/caffe2/operators/accumulate_op.h: In constructor ‘caffe2::AccumulateOp<T, Context>::AccumulateOp(const caffe2::OperatorDef&, caffe2::Workspace*)’:
/path/to/pytorch/caffe2/operators/accumulate_op.h:13:187: error: ‘GetSingleArgument<float>’ is not a member of ‘caffe2::AccumulateOp<T, Context>’
   AccumulateOp(const OperatorDef& operator_def, Workspace* ws)
                                                                                                                                                                                           ^                        
/path/to/pytorch/caffe2/operators/elementwise_ops.h: In constructor ‘caffe2::BinaryElementwiseWithArgsOp<InputTypes, Context, Functor, OutputTypeMap>::BinaryElementwiseWithArgsOp(const caffe2::OperatorDef&, caffe2::Workspace*)’:
/path/to/pytorch/caffe2/operators/elementwise_ops.h:106:189: error: ‘GetSingleArgument<bool>’ is not a member of ‘caffe2::BinaryElementwiseWithArgsOp<InputTypes, Context, Functor, OutputTypeMap>’
   BinaryElementwiseWithArgsOp(const OperatorDef& operator_def, Workspace* ws)
                                                                                                                                                                                             ^                       
/path/to/pytorch/caffe2/operators/elementwise_ops.h:106:272: error: ‘GetSingleArgument<int>’ is not a member of ‘caffe2::BinaryElementwiseWithArgsOp<InputTypes, Context, Functor, OutputTypeMap>’
   BinaryElementwiseWithArgsOp(const OperatorDef& operator_def, Workspace* ws)
                                                                                                                                                                                                                                                                                ^                      
/path/to/pytorch/caffe2/operators/elementwise_ops.h:106:350: error: ‘GetSingleArgument<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >’ is not a member of ‘caffe2::BinaryElementwiseWithArgsOp<InputTypes, Context, Functor, OutputTypeMap>’
   BinaryElementwiseWithArgsOp(const OperatorDef& operator_def, Workspace* ws)

Cause

I'm not sure. Could be that GetSingleArgument() is defined elsewhere?

Solution

Modify /path/to/densepose/detectron/ops/pool_points_interp.h. Change OperatorBase::GetSingleArgument<float> to this->template GetSingleArgument<float>

(Thanks to badpx@Github: https://github.com/facebookresearch/DensePose/pull/137/commits/51389c6a02173a25e9429825db452beb5e1cf3be)

2.8 fatal error: caffe2/utils/threadpool/ThreadPool.h: No such file or directory

Details

Occurs when running "make ops".

Main error message:

1	/path/to/pytorch/torch/lib/include/caffe2/core/workspace.h:19:48: fatal error: caffe2/utils/threadpool/ThreadPool.h: No such file or directory

This should only happen when your Caffe2 is installed with Anaconda.

Cause

If your Caffe2 is installed with Anaconda, these files may not be found anywhere in the Caffe2 directory, or in your hard disk at all.

Solution

In Anikily@Github's case, downloading Caffe2 source code and add its path to DensePose's include directories will work:

1	git clone [email protected]:pytorch/pytorch.git

and add one line in the beginning of DensePose/CMakeLists.txt:

1	include_directories("/path/to/pytorch")

The directory you include here should contain caffe2/utils/threadpool/ThreadPool.h and all the others.

I don't think this issue should be solved this way, but I'm sure that these files couldn't be found anywhere else. If anyone finds a better solution, please comment here to help the others.

2.9 Undefined symbol: _ZN6google8protobuf8internal9ArenaImpl28AllocateAlignedAndAddCleanupEmPFvPvE

Details

Occurred when running python detectron/tests/test_zero_even_op.py.

Main error message:

1	OSError: /path/to/densepose/build/libcaffe2_detectron_custom_ops_gpu.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl28AllocateAlignedAndAddCleanupEmPFvPvE

Cause

WTF is this!???
As can be seen, this symbol has something to do with Google, and protobuf.
I guess this is caused by a different protobuf version. Good news is that a proper version of protobuf was also built with Caffe2, so why not tell this to DensePose?

Solution

In /path/to/densepose/CMakeLists.txt, Add a few lines in the beginning:

add_library(libprotobuf STATIC IMPORTED) 

set(PROTOBUF_LIB "/path/to/pytorch/torch/lib/libprotobuf.a") 

set_property(TARGET libprotobuf PROPERTY IMPORTED_LOCATION "${PROTOBUF_LIB}")

You can find two target_link_libraries lines in this file(they are not adjacent):

1 2	target_link_libraries(caffe2_detectron_custom_ops caffe2_library) target_link_libraries(caffe2_detectron_custom_ops_gpu caffe2_gpu_library)

Edit the two lines, adding a "libprotobuf" at the end to each of them:

1 2	target_link_libraries(caffe2_detectron_custom_ops caffe2_library libprotobuf) target_link_libraries(caffe2_detectron_custom_ops_gpu caffe2_gpu_library libprotobuf)

Then run make ops again, and python detectron/tests/test_zero_even_op.py again.

(Thanks to hyounsamk@Github: https://github.com/facebookresearch/DensePose/issues/119)

After fixing this issue, my DensePose passed tests and was running flawlessly. If any more issues remain, don't hesitate to comment here~

2.10 Undefined symbol: _ZN6caffe219CPUOperatorRegistryB5cxx11Ev

Details

Occurred when running python detectron/tests/test_zero_even_op.py, with Caffe2 installed with Anaconda.

Main error message:

1	OSError: /path/to/densepose/build/libcaffe2_detectron_custom_ops_gpu.so: undefined symbol: _ZN6caffe219CPUOperatorRegistryB5cxx11Ev

Cause

As can be seen from the messy undefined symbol, this should have something to do with Caffe2 and probably CXX11(oh really???).

Run ldd -r /path/to/densepose/build/libcaffe2_detectron_custom_ops.so and the one or several undefined symbols with similar names will be shown, which should have been defined in libcaffe2.so. After running strings -a /path/to/pytorch/torch/lib/libcaffe2.so | grep _ZN6caffe219CPUOperator, a few similar symbols (two, in my case) would come up, but are different from the one undefined - "B5cxx11" is missing.

Why does DensePose want to find a symbol with "B5cxx11"? Who added this suffix?
It should be our GCC who did it when compiling DensePose with C++11 standard!

To find which version of GCC was Caffe2 built by, run strings -a /path/to/pytorch/torch/lib/libcaffe2.so | grep GCC:.
In my case, the output is:

1	GCC: (GNU) 4.9.2 20150212 (Red Hat 4.9.2-6)

Oh? It seems that Caffe2 developers are Red Hat lovers!
The Caffe2 installed with Anaconda was built by GCC 4.9.2, which had a slightly different standard on naming symbols.

Solution

The simpliest way out is to turn to GCC 4.9.2 for building DensePose, too.
Otherwise, maybe also consider compiling Caffe2/PyTorch from source code?

(Many thanks to Johnqczhang@Github: https://github.com/linkinpark213/linkinpark213.github.io/issues/12)

0 Motivation

Starting from this post, I decide to keep a record (tag: MineSweeping) of the issues I meet while working with environments and also their solutions.

Doing configurations in order to run others' code may be a difficult task, and is sometimes depressing, since various issues could arise, and the it's impossible for the authors to keep providing solutions for every user in the community. What's worse, after fixing some problems with a lot of struggle, one may have to waste the same amount of time on the same issue the next time he/she run it again. That's why I decide to keep this record: to avoid wasting time twice, while also helping others deal with problems if possible.