I'm trying to reconstruct surface/depthmap from normals using the frankot/chellappa algorithm. The rows and cols are the size of the img I'm trying to reconstruct the depth for.

I obtain the normal vectors like this:

rows, cols = imglist[0].shape
def find_NormAlbedo(sources, imglist, rows, cols):
    '''
    :param sources: a list of light source coordinates as [x,y,z] coordinates per light source
                    (shape (20,3) for 20 sources)
    :param imglist: a list of all images for one object
    :param rows: shape[0] of every image
    :param cols: shape[1] of every image
    :return: returns normals and albedo's for an object
    '''
    normal = np.zeros_like(imglist[0], dtype=np.ndarray)
    albedo = np.zeros_like(imglist[0])

    # for every pixel
    for x in range(rows):
        for y in range(cols):
            I = []  # intensity matrix of pixel x,y per image
            S = []  # lightsources
            for i in range(len(imglist)):
                img = imglist[i]
                I.append([img[x][y]])
                S.append(sources[i])

            # Least squares solution if S is invertible
            # pseudoinverse
            pseudoS = np.linalg.pinv(S)

            ntilde = pseudoS @ I
            p = np.linalg.norm(ntilde, ord=2)
            if p != 0.:
                n = ntilde / p
                n = n.flatten()
                # print(n)
                # print(n.shape)
            else:
                n = np.zeros_like(ntilde.flatten())

            normal[x][y] = n
            albedo[x][y] = p

    return normal, albedo

But suspect it's wrong because my albedo looks completely different from what I've seen in examples but have no clue where my mistake is...

Then I try to Get the surface from that using a wavepy function surface_to_grad:

def depthfromgradient(normalmap):
    '''
    :param normalmap: Previously obtained normals per pixel
    :return: Surface/Depth map from normalmap
    '''
    surfacep = np.zeros_like(normalmap)
    surfaceq = np.zeros_like(normalmap)
    for row in range(rows):
        for x in range(cols):
            #print(x)
            a, b, c = normalmap[row][x]
            #print(a, b, c)
            if c !=0:
                p = -a / c  # p=dZ/dx
                q = -b / c  # q=dZ/dy
                surfacep[row][x] = p
                surfaceq[row][x] = q
    return surface_from_grad.frankotchellappa(surfacep, surfaceq, reflec_pad=True)

My goal is to visualise the depthmap and the normalmap with cv.imshow(), but I'm not sure where I went wrong. These are my questions/ideas of where it went wrong:

-Is the albedomap plausible? If no, I think I misunderstood part of this algorithm.

-My depthmap has complex numbers, is this normal? Where do these come from?

-I looked at the shape of the normal map, the albedo map and the depth map, they all have shape (640, 500), yet I can only visualise the albedomap, the others give me the following error, what is the problem here?:

cv2.imshow('DepthMap', surface)
TypeError: Expected cv::UMat for argument 'mat'

Any help in narrowing down this problem would be welcome.

Note:I have tried converting everything to np arrays before using imshow().

CodePudding user response：

The problem is exactly as I described in my very first comment. The values you get for p (which become your albedo image) range from 0 to 1998.34. When you store that in a byte, you're just getting the low-order 8 bits, which wrap around.

If you change this:

            albedo[x][y] = p

to this:

            albedo[x,y] = p/8

you'll see that the resulting image looks great.

By the way, there are several optimizations you can do. Where you have xxx[x][y] with a numpy array, do xxx[x,y] instead. When you build your lightsources, instead of

            I = []  # intensity matrix of pixel x,y per image
            S = []  # lightsources
            for i in range(len(imglist)):
                img = imglist[i]
                I.append([img[x][y]])
                S.append(sources[i])

do

            I = [img[x,y] for img in imglist]
            S = sources[:len(imglist),:]

and your obtainData function can be made more readable by doing:

    # Read all images
    paths = (
     fr".\PSData\PSData\{things[i]}\Objects\Image_01.png",
     fr".\PSData\PSData\{things[i]}\Objects\Image_02.png",
     fr".\PSData\PSData\{things[i]}\Objects\Image_03.png",
     fr".\PSData\PSData\{things[i]}\Objects\Image_04.png",
     fr".\PSData\PSData\{things[i]}\Objects\Image_05.png"
    )

    imgs = [cv2.imread(p,0) for p in paths]
...
    # Apply masks to images: cv2.bitwise
    imglist = [cv2.bitwise_or(img, img, mask=mask(img, threshold)) for img in imgs]