The first one should have worked for you.
[group] (workspace=[current])
[app] (name=libreoffice) (class=libreoffice-writer)
[app] (name=libreoffice) (class=libreoffice-calc)
[Position] (UPPERLEFT) {4% 3%}
[Shaded] {no}
[Deco] {NORMAL}
[Sticky] {no}
[Minimized] {no}
[Maximized] {no}
[Fullscreen] {no}
[Layer] {8}
[Alpha] {255}
[end]

Normally you do not have to restart fluxbox for changes in the apps file to take effect, but you can try that.
How are you opening your libreoffice documents? Only other thing that comes to mind is that libreoffice is also assigning a unique role to your documents and you may need to account for that in the apps file.
Open a couple of documents in your usual way and then right click on the title bar to get the window menu. Use the Remember option to assign some per-app conditions. After that, open the ~/.fluxbox/apps file and how fluxbox defined the name, class, and role (if any). Post the results, please.
Are your other apps grouping, sizing etc ...? Proof read your ~/.fluxbox/apps file - if I remember correctly, if there is an error in the file (like a missing [end]) then fluxbox stops there, so the applications listed before the error behave as expected, anything after the error is not read.
Edit Number 1 - This is new -
I created a couple of documents and then opened them up to assign some per-app settings; got this line:
[app] (name=libreoffice) (class=LibreOffice 5.0)
Apparently if the document is not blank, it is assigned a different class name. Adding that line to the groups stanza fixes the issue so blank and existing docs/spreadsheets group.
[group] (workspace=[current])
[app] (name=libreoffice) (class=libreoffice-writer)
[app] (name=libreoffice) (class=libreoffice-calc)
[app] (name=libreoffice) (class=LibreOffice 5.0)
[Position] (UPPERLEFT) {4% 3%}
[Shaded] {no}
[Deco] {NORMAL}
[Sticky] {no}
[Minimized] {no}
[Maximized] {no}
[Fullscreen] {no}
[Layer] {8}
[Alpha] {255}
[end]