GNU assembler on Windows: WriteFile returns ERROR_INVALID

Until now, I've always written Assembly-code in GAS Syntax on Linux, so I wanted to try out how it'd be on Windows. The first goal was to print a single character onto stdout - which didn't work

Here's my code:

.intel_syntax noprefix

.extern GetStdHandle
.extern GetLastError
.extern WriteFile
.extern ExitProcess

.section .rodata
        .Lchar: .ascii "F"

.section .data
    .Lbytes_written: .long 0

.section .text

.global main

main:
        mov rcx, -11 //-11 = stdout
        call GetStdHandle
        mov rcx,  rax
        lea rdx, [rip   .Lchar]
        mov  r8, 1
        lea r9, [rip   .Lbytes_written]
        push 0
        call WriteFile
        call GetLastError //After this call, rax=0x6

        xor rcx, rcx
        call ExitProcess

After compiling it using gcc -g -o example.exe ./example.S (I use mingw64), nothing is printed. When stepping through the code with a debugger, I noticed that GetStdHandle doesn't fail (calling GetLastError directly after it returned 0), but WriteFile fails with 0x6, which is ERROR_INVALID_HANDLE.

So my question is: What could be the issue here? This question is probably very stupid to some, so I apologize in advance. Thank you!

Edit: This has gotten weirder than I thought. Take these two blocks of code

.section .rodata
.Lchar: .ascii "F"

.section .data
.Lbytes_written: .long 0

.section .text

.global main

main:
    sub rsp, 8
    mov rcx, -11
    call GetStdHandle
    mov rcx, rax
    lea rdx, [rip   .Lchar]
    mov r8, 1
    lea r9, [rip   .Lbytes_written]
    push 0
    push 0
    call WriteFile
    call GetLastError
    add rsp, 16
    add rsp, 8
    ret

.section .rodata
.Lchar: .ascii "F"

.section .data
.Lbytes_written: .long 0

.section .text

.global main

main:
    sub rsp, 24
    mov rcx, -11
    call GetStdHandle
    mov rcx, rax
    lea rdx, [rip   .Lchar]
    mov r8, 1
    lea r9, [rip   .Lbytes_written]
    push 0
    push 0
    call WriteFile
    call GetLastError
    add rsp, 16
    add rsp, 24
    ret

The only difference is that the second code block allocates 16B more on the Stack, which shouldn't be a problem, as all calls are still 16B-aligned. Yet for the second code block, the call to WriteFile does not work. Funnily enough, both function calls to GetStdHandle succeed and return the same value (which is 84 in my case) What could be the source of that?

CodePudding user response：

main:
    sub rsp, 8
    mov rcx, -11
    call GetStdHandle

Memory trashed at this point. The top four slots (32 bytes) in the stack need to be free. The correct line seems to be

    sub rsp, 40

Moving right along:

    call WriteFile
    call GetLastError

Mistake here. Check rax for error code. If no error, GetLastError returns the previous error whatever it was.

If something seems absolutely nuts, check code further along. As far as I can determine from trying to make this stuff work, Windows disassembles your code in some places and code that isn't reached can still crash if it would misalign the stack. I had a case where the crash was due to the difference between not commenting code and commenting code that was unconditionally jumped around.

CodePudding user response：

So I'll post the solution as an answer so that other can find it more quickly - thank you to Peter Cordes, Joshua and RbMm for helping me on this one!

So the main issue with my code was that I violated Windows' ABI, which included

not aligning the stack properly before a function call
not adding a 32B padding onto the stack, meaning that the 5th parameter was incorrectly provided

Here's a working version of this code:

.intel_syntax noprefix

.extern GetStdHandle
.extern GetLastError
.extern WriteFile

.section .rodata
.Lchar: .ascii "F"

.section .data
.Lbytes_written: .long 0

.section .text

.global main

main:
    sub rsp, 40
    mov rcx, -11
    call GetStdHandle
    mov rcx, rax
    lea rdx, [rip   .Lchar]
    mov r8, 1
    lea r9, [rip   .Lbytes_written]
    mov QWORD PTR [rsp   32], 0
    call WriteFile
    add rsp, 40
    ret